[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:10 dist-test-slave-r9x1 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:10 dist-test-slave-r9x1 krb5kdc[399](info): set up 2 sockets
Sep 27 21:12:10 dist-test-slave-r9x1 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 21:12:11.839627 392 mini_kdc.cc:121] Time spent starting KDC: real 2.090s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 21:12:12.001758 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.162s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 21:12:12.291776 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.290s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 21:12:12.525451 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.234s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/test-user.keytab.
W20260927 21:12:12.974756 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.449s user 0.001s sys 0.000s
Sep 27 21:12:12 dist-test-slave-r9x1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:13.020953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:13.103219 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:13.103472 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:13.103494 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:13.104768 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:13.104833 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:13.104847 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:13.104859 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:13.104871 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:13.106580 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:13.106868 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:13.107124 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:12:13.113009 412 server_base.cc:1034] running on GCE node
W20260927 21:12:13.112968 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:13.112986 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:13.112972 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:13.113480 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:13.113809 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:13.114995 412 hybrid_clock.cc:648] HybridClock initialized: now 1790543533114974 us; error 37 us; skew 500 ppm
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:13.117818 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:13.118335 412 webserver.cc:460] Webserver started at http://127.0.98.62:36219/ using document root <none> and password file <none>
I20260927 21:12:13.118593 412 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:13.118646 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:13.118777 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:13.119894 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "6e6f5269d41a4fa59fecee7892331e63"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "8d83224a0c078bedf7c3c099acb0eb5d"
server_key_iv: "7ef66579cccfd5938e0adca9ae530560"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.120285 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e6f5269d41a4fa59fecee7892331e63"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "8d83224a0c078bedf7c3c099acb0eb5d"
server_key_iv: "7ef66579cccfd5938e0adca9ae530560"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.121707 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:13.122545 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.122735 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:13.122818 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "6e6f5269d41a4fa59fecee7892331e63"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "8d83224a0c078bedf7c3c099acb0eb5d"
server_key_iv: "7ef66579cccfd5938e0adca9ae530560"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.122893 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:13.131937 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:13.134225 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:13.134363 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:13.138087 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44765
I20260927 21:12:13.138127 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44765 every 8 connection(s)
I20260927 21:12:13.138476 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:13.139382 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:13.142020 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63: Bootstrap starting.
I20260927 21:12:13.142635 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:13.142903 480 log.cc:824] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:13.143784 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63: No bootstrap required, opened a new log
I20260927 21:12:13.145577 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f5269d41a4fa59fecee7892331e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } }
I20260927 21:12:13.145789 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:13.145829 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6f5269d41a4fa59fecee7892331e63, State: Initialized, Role: FOLLOWER
I20260927 21:12:13.145810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 412
I20260927 21:12:13.145900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:13.145988 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f5269d41a4fa59fecee7892331e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } }
I20260927 21:12:13.146083 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:13.146118 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:13.146162 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:13.146728 392 external_mini_cluster.cc:1428] Setting key a7a90860262da1c7dde9eab3869ac177
I20260927 21:12:13.146977 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f5269d41a4fa59fecee7892331e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } }
I20260927 21:12:13.147095 480 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e6f5269d41a4fa59fecee7892331e63; no voters:
I20260927 21:12:13.147324 480 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:13.147392 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:13.147603 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:13.147611 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [term 1 LEADER]: Becoming Leader. State: Replica: 6e6f5269d41a4fa59fecee7892331e63, State: Running, Role: LEADER
I20260927 21:12:13.148003 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f5269d41a4fa59fecee7892331e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } }
I20260927 21:12:13.148386 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6f5269d41a4fa59fecee7892331e63. Latest consensus state: current_term: 1 leader_uuid: "6e6f5269d41a4fa59fecee7892331e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f5269d41a4fa59fecee7892331e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } } }
I20260927 21:12:13.148396 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e6f5269d41a4fa59fecee7892331e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f5269d41a4fa59fecee7892331e63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44765 } } }
I20260927 21:12:13.148469 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:13.148491 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:13.148665 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:13.149202 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:13.151016 494 catalog_manager.cc:1347] Generated new cluster ID: 616d99ca9eb34baa87c626bc363bbb58
I20260927 21:12:13.151079 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:13.162127 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:13.162804 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:13.147790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.0.98.62:44765)
0927 21:12:13.147942 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:13.148009 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:13.148425 (+ 416us) server_negotiation.cc:407] Connection header received
0927 21:12:13.150873 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:13.150878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:13.150932 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:13.151072 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:13.152594 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.153649 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.154869 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.155360 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.155484 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:13.158969 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:13.158973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:13.158974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:13.158988 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:13.160605 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.161064 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.161067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.161067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.161165 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.161464 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.161465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.161465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.161741 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:13.161908 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:13.162313 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 21:12:13.162452 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260927 21:12:13.162858 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:13.169813 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6f5269d41a4fa59fecee7892331e63: Generated new TSK 0
I20260927 21:12:13.170045 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:13.196854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44765
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:13.280961 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:13.281189 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:13.281224 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:13.282585 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:13.282639 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:13.282680 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:13.284277 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44765
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:13.284538 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:13.284756 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:13.289832 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:13.289896 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:13.289836 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:13.290089 506 server_base.cc:1034] running on GCE node
I20260927 21:12:13.290331 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:13.290596 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:13.291755 506 hybrid_clock.cc:648] HybridClock initialized: now 1790543533291733 us; error 35 us; skew 500 ppm
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:13.294425 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:13.294885 506 webserver.cc:460] Webserver started at http://127.0.98.1:33737/ using document root <none> and password file <none>
I20260927 21:12:13.295073 506 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:13.295110 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:13.295202 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:13.296226 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "47bb8d7a1ae64ccdb795e959aa758271"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "51d67cac92be95947113c06311ae6049"
server_key_iv: "43e0495c06518a36875b3c8d31c50a0f"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.296546 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47bb8d7a1ae64ccdb795e959aa758271"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "51d67cac92be95947113c06311ae6049"
server_key_iv: "43e0495c06518a36875b3c8d31c50a0f"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.297825 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:13.298530 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.298674 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:12:13.298753 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "47bb8d7a1ae64ccdb795e959aa758271"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "51d67cac92be95947113c06311ae6049"
server_key_iv: "43e0495c06518a36875b3c8d31c50a0f"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.298823 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:13.315303 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:13.317637 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:13.317760 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:13.318010 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:13.318316 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:13.318356 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.318379 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:13.318403 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.325063 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34279
I20260927 21:12:13.325112 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34279 every 8 connection(s)
I20260927 21:12:13.325466 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:13.332279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 506
I20260927 21:12:13.332360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:13.332509 392 external_mini_cluster.cc:1428] Setting key 7bfc5686b894bfbe5b39ea493b844a63
I20260927 21:12:13.336061 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:13.326199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60825 (local address 127.0.98.62:44765)
0927 21:12:13.326280 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:13.326283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:13.326429 (+ 146us) server_negotiation.cc:407] Connection header received
0927 21:12:13.326976 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:13.326977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:13.326990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:13.327053 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:13.328016 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.328543 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.329633 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.330259 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.330334 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:13.332804 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:13.332806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:13.332806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:13.332815 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:13.334508 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.334983 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.334984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.334985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.335051 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.335408 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.335409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.335409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.335526 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:13.335574 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:13.335916 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 21:12:13.335991 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:12:13.336338 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:13.326148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44765 (local address 127.0.98.1:60825)
0927 21:12:13.326311 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:13.326350 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:13.326861 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:13.327122 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:13.327125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:13.327706 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:13.327920 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:13.327925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.328652 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.328653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:13.329520 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:13.329523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.330379 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.330381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:13.330511 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:13.331134 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:13.331144 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:13.332705 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:13.334620 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:13.334629 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:13.334631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:13.334913 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:13.335147 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:13.335148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:13.335149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:13.335332 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:13.335583 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:13.335589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:13.335782 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 21:12:13.335901 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 21:12:13.336101 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260927 21:12:13.336838 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44765
I20260927 21:12:13.336959 635 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:13.337167 635 heartbeater.cc:511] Master 127.0.98.62:44765 requested a full tablet report, sending...
I20260927 21:12:13.337728 444 ts_manager.cc:194] Registered new tserver with Master: 47bb8d7a1ae64ccdb795e959aa758271 (127.0.98.1:34279)
I20260927 21:12:13.338358 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60825
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:13.358220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44765
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:13.441138 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:13.441452 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:13.441489 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:13.442868 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:13.442940 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:13.442983 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:13.444700 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44765
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:13.445016 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:13.445283 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:12:13.451082 640 server_base.cc:1034] running on GCE node
W20260927 21:12:13.451045 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:13.451063 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:13.451045 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:13.451575 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:13.451908 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:13.453082 640 hybrid_clock.cc:648] HybridClock initialized: now 1790543533453063 us; error 39 us; skew 500 ppm
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:13.455889 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:13.456414 640 webserver.cc:460] Webserver started at http://127.0.98.2:42165/ using document root <none> and password file <none>
I20260927 21:12:13.456652 640 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:13.456704 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:13.456836 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:13.457976 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "e760599af0024f98833bd73e52eead3f"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "dac0a58905d3cb095ba0ba116ad170f5"
server_key_iv: "e56a8ba87c5a296c13b0d272147f6d06"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.458324 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e760599af0024f98833bd73e52eead3f"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "dac0a58905d3cb095ba0ba116ad170f5"
server_key_iv: "e56a8ba87c5a296c13b0d272147f6d06"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.459723 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:12:13.460487 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.460690 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:13.460789 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "e760599af0024f98833bd73e52eead3f"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "dac0a58905d3cb095ba0ba116ad170f5"
server_key_iv: "e56a8ba87c5a296c13b0d272147f6d06"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.460875 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:13.476629 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:13.478878 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:13.479027 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:13.479250 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:13.479570 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:13.479604 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.479632 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:13.479648 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.486137 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41781
I20260927 21:12:13.486168 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41781 every 8 connection(s)
I20260927 21:12:13.486508 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:13.493180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 640
I20260927 21:12:13.493299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:13.493475 392 external_mini_cluster.cc:1428] Setting key f0ea8fa32ff9e123718a903b40fb5adf
I20260927 21:12:13.496093 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:13.487266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53621 (local address 127.0.98.62:44765)
0927 21:12:13.487334 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:13.487336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:13.487530 (+ 194us) server_negotiation.cc:407] Connection header received
0927 21:12:13.488005 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:13.488006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:13.488016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:13.488077 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:13.489037 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.489579 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.490597 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.490788 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.490840 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:13.492998 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:13.493000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:13.493000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:13.493008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:13.494621 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.495050 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.495051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.495051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.495136 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.495431 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.495432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.495433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.495563 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:13.495624 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:13.495907 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 21:12:13.496001 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:12:13.496285 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:13.487247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44765 (local address 127.0.98.2:53621)
0927 21:12:13.487402 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:13.487446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:13.487928 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:13.488143 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:13.488147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:13.488750 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:13.488959 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:13.488966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.489676 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.489677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:13.490517 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:13.490520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.490848 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.490848 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:13.490953 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:13.491437 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:13.491446 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:13.492856 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:13.494716 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:13.494723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:13.494725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:13.494981 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:13.495197 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:13.495198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:13.495198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:13.495329 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:13.495624 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:13.495631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:13.495782 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 21:12:13.495877 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:12:13.496068 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260927 21:12:13.496734 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44765
I20260927 21:12:13.496838 769 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:13.497044 769 heartbeater.cc:511] Master 127.0.98.62:44765 requested a full tablet report, sending...
I20260927 21:12:13.497448 444 ts_manager.cc:194] Registered new tserver with Master: e760599af0024f98833bd73e52eead3f (127.0.98.2:41781)
I20260927 21:12:13.497876 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53621
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:13.517513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44765
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:13.600596 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:13.600884 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:13.600931 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:13.602304 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:13.602355 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:13.602393 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:13.604197 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44765
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:13.604501 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:13.604763 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:13.611179 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:13.611251 774 server_base.cc:1034] running on GCE node
W20260927 21:12:13.611172 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:13.611174 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:13.611624 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:13.611867 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:13.613071 774 hybrid_clock.cc:648] HybridClock initialized: now 1790543533613052 us; error 36 us; skew 500 ppm
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:13.615899 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:13.616429 774 webserver.cc:460] Webserver started at http://127.0.98.3:39119/ using document root <none> and password file <none>
I20260927 21:12:13.616643 774 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:13.616694 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:13.616825 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:13.617964 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "02d379d4335c497b8656ccde23d8ace5"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "7a96bb1dc5fa5fbe0cf95f139e203fab"
server_key_iv: "6ef93b213d1e958899e46dd08b1f0479"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.618304 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02d379d4335c497b8656ccde23d8ace5"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "7a96bb1dc5fa5fbe0cf95f139e203fab"
server_key_iv: "6ef93b213d1e958899e46dd08b1f0479"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.619814 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:13.620599 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.620774 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:13.620857 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "02d379d4335c497b8656ccde23d8ace5"
format_stamp: "Formatted at 2026-09-27 21:12:13 on dist-test-slave-r9x1"
server_key: "7a96bb1dc5fa5fbe0cf95f139e203fab"
server_key_iv: "6ef93b213d1e958899e46dd08b1f0479"
server_key_version: "encryptionkey@0"
I20260927 21:12:13.620919 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:13.640214 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:13.642568 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:13.642712 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:13.642937 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:13.643265 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:13.643299 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.643328 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:13.643350 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:13.650357 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35873
I20260927 21:12:13.650403 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35873 every 8 connection(s)
I20260927 21:12:13.650743 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:13.652117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 774
I20260927 21:12:13.652204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:13.652362 392 external_mini_cluster.cc:1428] Setting key 50bc9137efd0759426d37539b40a1581
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:13.661218 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:13.651438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40659 (local address 127.0.98.62:44765)
0927 21:12:13.651515 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:13.651518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:13.651676 (+ 158us) server_negotiation.cc:407] Connection header received
0927 21:12:13.652198 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:13.652200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:13.652212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:13.652320 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:13.653590 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.654213 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.655331 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.655555 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.655620 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:13.658497 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:13.658499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:13.658499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:13.658506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:13.659932 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.660360 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.660361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.660361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.660417 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.660665 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.660666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.660667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.660790 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:13.660845 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:13.661087 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 21:12:13.661151 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:12:13.661476 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:13.651414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44765 (local address 127.0.98.3:40659)
0927 21:12:13.651556 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:13.651594 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:13.652096 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:13.652411 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:13.652417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:13.653111 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:13.653394 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:13.653401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.654339 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.654341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:13.655214 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:13.655217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.655800 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.655803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:13.656036 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:13.656797 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:13.656808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:13.658346 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:13.660036 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:13.660044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:13.660046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:13.660288 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:13.660472 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:13.660473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:13.660473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:13.660598 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:13.660844 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:13.660848 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:13.660984 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:12:13.661067 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:12:13.661202 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260927 21:12:13.661966 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44765
I20260927 21:12:13.662057 903 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:13.662227 903 heartbeater.cc:511] Master 127.0.98.62:44765 requested a full tablet report, sending...
I20260927 21:12:13.662611 444 ts_manager.cc:194] Registered new tserver with Master: 02d379d4335c497b8656ccde23d8ace5 (127.0.98.3:35873)
I20260927 21:12:13.662994 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40659
I20260927 21:12:13.664779 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:12:13.687836 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:13.680349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.0.98.1:34279)
0927 21:12:13.680539 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:13.680544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:13.680554 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:13.680579 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:13.680580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:13.680593 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:13.680657 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:13.681233 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.682004 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.682723 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.683043 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.683104 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:13.685242 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:13.685251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:13.685251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:13.685260 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:13.686721 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.687052 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.687055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.687056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.687110 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.687287 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.687288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.687288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.687446 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:13.687583 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:13.687652 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:12:13.687731 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
W20260927 21:12:13.688241 548 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42364
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:12:13.709219 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:13.702540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.0.98.1:34279)
0927 21:12:13.702594 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:13.702597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:13.702667 (+ 70us) server_negotiation.cc:407] Connection header received
0927 21:12:13.702736 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:13.702737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:13.702749 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:13.702798 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:13.703347 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.703845 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.704544 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:13.704749 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:13.704818 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:13.706934 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:13.706936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:13.706936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:13.706945 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:13.708198 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.708533 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.708534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.708535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.708589 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:13.708746 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:13.708747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:13.708747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:13.708848 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:13.708979 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:13.709040 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:12:13.709125 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:12:13.709800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 506
I20260927 21:12:13.714020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 640
I20260927 21:12:13.718900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 774
I20260927 21:12:13.722550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 412
2026-09-27T21:12:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3998 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[917](info): set up 2 sockets
Sep 27 21:12:13 dist-test-slave-r9x1 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 21:12:14.672621 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:14 dist-test-slave-r9x1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:14.793210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:14.874541 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:14.874785 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:14.874818 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:14.876051 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:14.876101 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:14.876116 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:14.876127 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:14.876137 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:14.877758 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:14.877996 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:14.878183 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:12:14.883111 930 server_base.cc:1034] running on GCE node
W20260927 21:12:14.883204 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:14.883314 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:14.883165 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:14.883591 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:14.883906 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:14.885069 930 hybrid_clock.cc:648] HybridClock initialized: now 1790543534885054 us; error 35 us; skew 500 ppm
Sep 27 21:12:14 dist-test-slave-r9x1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:14.887745 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:14.888203 930 webserver.cc:460] Webserver started at http://127.0.98.62:36331/ using document root <none> and password file <none>
I20260927 21:12:14.888415 930 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:14.888463 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:14.888584 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:14.889792 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "c123629a27d940fe97025d24abd29914"
format_stamp: "Formatted at 2026-09-27 21:12:14 on dist-test-slave-r9x1"
server_key: "e905ea1219de7c780ce3ffa601a9646e"
server_key_iv: "32070e498efb304ec481c7f5a3a4f0dd"
server_key_version: "encryptionkey@0"
I20260927 21:12:14.890125 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "c123629a27d940fe97025d24abd29914"
format_stamp: "Formatted at 2026-09-27 21:12:14 on dist-test-slave-r9x1"
server_key: "e905ea1219de7c780ce3ffa601a9646e"
server_key_iv: "32070e498efb304ec481c7f5a3a4f0dd"
server_key_version: "encryptionkey@0"
I20260927 21:12:14.891455 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:14.892180 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:14.892374 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:14.892462 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "c123629a27d940fe97025d24abd29914"
format_stamp: "Formatted at 2026-09-27 21:12:14 on dist-test-slave-r9x1"
server_key: "e905ea1219de7c780ce3ffa601a9646e"
server_key_iv: "32070e498efb304ec481c7f5a3a4f0dd"
server_key_version: "encryptionkey@0"
I20260927 21:12:14.892525 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:14.901820 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:14.903959 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:14.904111 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:14.907974 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33241
I20260927 21:12:14.908032 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33241 every 8 connection(s)
I20260927 21:12:14.908370 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:14.909256 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:14.911803 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914: Bootstrap starting.
I20260927 21:12:14.912423 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:14.912708 998 log.cc:824] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:14.913482 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914: No bootstrap required, opened a new log
I20260927 21:12:14.915192 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123629a27d940fe97025d24abd29914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } }
I20260927 21:12:14.915362 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:14.915387 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c123629a27d940fe97025d24abd29914, State: Initialized, Role: FOLLOWER
I20260927 21:12:14.915531 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123629a27d940fe97025d24abd29914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } }
I20260927 21:12:14.915603 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:14.915629 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:14.915674 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:14.916250 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123629a27d940fe97025d24abd29914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } }
I20260927 21:12:14.916350 998 leader_election.cc:302] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c123629a27d940fe97025d24abd29914; no voters:
I20260927 21:12:14.916530 998 leader_election.cc:288] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:14.916636 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:14.916785 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:14.917022 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [term 1 LEADER]: Becoming Leader. State: Replica: c123629a27d940fe97025d24abd29914, State: Running, Role: LEADER
I20260927 21:12:14.917126 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123629a27d940fe97025d24abd29914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } }
I20260927 21:12:14.917511 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c123629a27d940fe97025d24abd29914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123629a27d940fe97025d24abd29914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } } }
I20260927 21:12:14.917536 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c123629a27d940fe97025d24abd29914. Latest consensus state: current_term: 1 leader_uuid: "c123629a27d940fe97025d24abd29914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c123629a27d940fe97025d24abd29914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33241 } } }
I20260927 21:12:14.917627 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:14.917588 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:14.918023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 930
I20260927 21:12:14.918102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:14.917878 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:14.918255 392 external_mini_cluster.cc:1428] Setting key c32fc03833f4565226c9d58c2b834e44
I20260927 21:12:14.918736 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:14.920472 1010 catalog_manager.cc:1347] Generated new cluster ID: 376e76c47a9e4a7d900e9be782e1af9e
I20260927 21:12:14.920531 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:12:14 dist-test-slave-r9x1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:14.930099 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:14.918721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.0.98.62:33241)
0927 21:12:14.918997 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:14.919006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:14.919034 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:12:14.919137 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:14.919143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:14.919188 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:14.919304 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:14.920018 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:14.921301 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:14.922401 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:14.923045 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:14.923161 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:14.926045 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:14.926051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:14.926052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:14.926066 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:14.928008 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:14.928463 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:14.928467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:14.928467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:14.928582 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:14.928780 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:14.928781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:14.928781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:14.929086 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:14.929232 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:14.929662 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 21:12:14.929838 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":64,"threads_started":1}
I20260927 21:12:14.938226 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:14.938988 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:14.955353 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c123629a27d940fe97025d24abd29914: Generated new TSK 0
I20260927 21:12:14.955585 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:14.990561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33241
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:15.074849 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:15.075076 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:15.075111 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:15.076350 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:15.076411 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:15.076447 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:15.078028 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33241
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:15.078315 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:15.078544 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:15.083766 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:15.083775 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:15.083787 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:15.084012 1024 server_base.cc:1034] running on GCE node
I20260927 21:12:15.084210 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:15.084437 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:15.085609 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790543535085572 us; error 54 us; skew 500 ppm
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:15.088268 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:15.088786 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41905/ using document root <none> and password file <none>
I20260927 21:12:15.089016 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:15.089066 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:15.089190 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:15.090334 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "84879e8a7111448f9085d4b9f209274b"
format_stamp: "Formatted at 2026-09-27 21:12:15 on dist-test-slave-r9x1"
server_key: "6b62405096fcf2327ff8d204e5b049f1"
server_key_iv: "de292791db4bb96ec77e70debfd44260"
server_key_version: "encryptionkey@0"
I20260927 21:12:15.090684 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "84879e8a7111448f9085d4b9f209274b"
format_stamp: "Formatted at 2026-09-27 21:12:15 on dist-test-slave-r9x1"
server_key: "6b62405096fcf2327ff8d204e5b049f1"
server_key_iv: "de292791db4bb96ec77e70debfd44260"
server_key_version: "encryptionkey@0"
I20260927 21:12:15.092059 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:15.093052 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:15.093246 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:15.093375 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "84879e8a7111448f9085d4b9f209274b"
format_stamp: "Formatted at 2026-09-27 21:12:15 on dist-test-slave-r9x1"
server_key: "6b62405096fcf2327ff8d204e5b049f1"
server_key_iv: "de292791db4bb96ec77e70debfd44260"
server_key_version: "encryptionkey@0"
I20260927 21:12:15.093452 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:15.109844 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:15.111999 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:15.112120 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:15.112357 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:15.112680 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:15.112716 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:15.112737 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:15.112749 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:15.118567 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42987
I20260927 21:12:15.118628 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42987 every 8 connection(s)
I20260927 21:12:15.118943 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:15.125983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 1024
I20260927 21:12:15.126080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:15.126235 392 external_mini_cluster.cc:1428] Setting key 41486a7abcd6d81855d2f82ecf9a63db
I20260927 21:12:15.128410 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.119652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52081 (local address 127.0.98.62:33241)
0927 21:12:15.119719 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.119721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.119923 (+ 202us) server_negotiation.cc:407] Connection header received
0927 21:12:15.120899 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.120901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.120916 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.120963 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:15.121643 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.122164 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.123165 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.123390 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.123447 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.125614 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:15.125616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:15.125616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:15.125629 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:15.127057 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.127462 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.127464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.127464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.127524 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.127746 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.127747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.127747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.127864 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:15.127920 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.128198 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.128292 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:12:15.128566 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.119622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33241 (local address 127.0.98.1:52081)
0927 21:12:15.119784 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:15.119822 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:15.120774 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:15.121038 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:15.121041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:15.121342 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:15.121546 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.121550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.122256 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.122256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.123081 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.123083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.123460 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.123461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.123582 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.124099 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:15.124107 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:15.125496 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:15.127132 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:15.127141 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:15.127144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:15.127390 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:15.127577 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:15.127577 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:15.127578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:15.127691 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:15.127919 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:15.127926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:15.128074 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 21:12:15.128170 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:12:15.128339 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260927 21:12:15.129220 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33241
I20260927 21:12:15.129349 1153 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:15.129535 1153 heartbeater.cc:511] Master 127.0.98.62:33241 requested a full tablet report, sending...
I20260927 21:12:15.130115 962 ts_manager.cc:194] Registered new tserver with Master: 84879e8a7111448f9085d4b9f209274b (127.0.98.1:42987)
I20260927 21:12:15.130724 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52081
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:15.152765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33241
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:15.235342 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:15.235605 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:15.235646 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:15.237144 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:15.237203 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:15.237244 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:15.238927 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33241
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:15.239209 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:15.239454 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:15.244495 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:15.244468 1158 server_base.cc:1034] running on GCE node
W20260927 21:12:15.244445 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:15.244427 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:15.244918 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:15.245219 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:15.246421 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790543535246406 us; error 30 us; skew 500 ppm
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:15.249035 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:15.249512 1158 webserver.cc:460] Webserver started at http://127.0.98.2:46005/ using document root <none> and password file <none>
I20260927 21:12:15.249737 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:15.249785 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:15.249909 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:15.250972 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "8f8cea4078a744fe892e496d748181d8"
format_stamp: "Formatted at 2026-09-27 21:12:15 on dist-test-slave-r9x1"
server_key: "b71511e54d7e2b03b1de104c55c5e8e3"
server_key_iv: "573f2e0fcd916da8f59f0f9b10e8991c"
server_key_version: "encryptionkey@0"
I20260927 21:12:15.251323 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8f8cea4078a744fe892e496d748181d8"
format_stamp: "Formatted at 2026-09-27 21:12:15 on dist-test-slave-r9x1"
server_key: "b71511e54d7e2b03b1de104c55c5e8e3"
server_key_iv: "573f2e0fcd916da8f59f0f9b10e8991c"
server_key_version: "encryptionkey@0"
I20260927 21:12:15.252897 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:15.253918 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:15.254114 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:15.254212 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "8f8cea4078a744fe892e496d748181d8"
format_stamp: "Formatted at 2026-09-27 21:12:15 on dist-test-slave-r9x1"
server_key: "b71511e54d7e2b03b1de104c55c5e8e3"
server_key_iv: "573f2e0fcd916da8f59f0f9b10e8991c"
server_key_version: "encryptionkey@0"
I20260927 21:12:15.254287 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:15.266681 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:15.268935 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:15.269086 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:15.269351 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:15.269673 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:15.269709 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:15.269737 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:15.269755 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:15.275935 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38271
I20260927 21:12:15.276000 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38271 every 8 connection(s)
I20260927 21:12:15.276309 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:12:15.278091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 1158
I20260927 21:12:15.278195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:15.278362 392 external_mini_cluster.cc:1428] Setting key 9d3f3bcf675401299bf43a667fefc2c9
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:15.286523 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.277056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53481 (local address 127.0.98.62:33241)
0927 21:12:15.277138 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.277141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.277343 (+ 202us) server_negotiation.cc:407] Connection header received
0927 21:12:15.277888 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.277889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.277898 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.277945 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:15.279121 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.279676 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.280688 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.280891 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.280954 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.283170 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:15.283172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:15.283172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:15.283181 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:15.284992 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.285499 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.285501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.285501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.285567 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.285806 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.285807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.285808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.285932 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:15.285989 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.286281 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.286379 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:12:15.286646 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.277029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33241 (local address 127.0.98.2:53481)
0927 21:12:15.277188 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:15.277228 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:15.277781 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:15.278046 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:15.278052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:15.278763 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:15.279031 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.279038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.279763 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.279764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.280603 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.280606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.280963 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.280964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.281093 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.281647 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:15.281657 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:15.283064 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:15.285101 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:15.285111 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:15.285114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:15.285419 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:15.285624 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:15.285625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:15.285626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:15.285740 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:15.285989 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:15.285996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:15.286150 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 21:12:15.286237 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:12:15.286414 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260927 21:12:15.287281 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33241
I20260927 21:12:15.287474 1287 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:15.287868 1287 heartbeater.cc:511] Master 127.0.98.62:33241 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:12:15.288398 962 ts_manager.cc:194] Registered new tserver with Master: 8f8cea4078a744fe892e496d748181d8 (127.0.98.2:38271)
I20260927 21:12:15.288784 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:15.309075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33241
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:15.388940 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:15.389169 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:15.389196 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:15.390497 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:15.390552 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:15.390581 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:15.392180 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33241
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:15.392441 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:15.392683 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:15.397632 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:15.397681 1292 server_base.cc:1034] running on GCE node
W20260927 21:12:15.397693 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:15.397632 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:15.398002 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:15.398240 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:15.399416 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790543535399404 us; error 34 us; skew 500 ppm
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:15.402096 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:15.402595 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37015/ using document root <none> and password file <none>
I20260927 21:12:15.402822 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:15.402882 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:15.403010 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:15.404072 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "5bef81cda84b486e9c1f941a408a0be5"
format_stamp: "Formatted at 2026-09-27 21:12:15 on dist-test-slave-r9x1"
server_key: "49c8e52a8c8a637f79f7b5d28121d9f4"
server_key_iv: "1b279fc4591cc2168983259c44cc9da1"
server_key_version: "encryptionkey@0"
I20260927 21:12:15.404407 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5bef81cda84b486e9c1f941a408a0be5"
format_stamp: "Formatted at 2026-09-27 21:12:15 on dist-test-slave-r9x1"
server_key: "49c8e52a8c8a637f79f7b5d28121d9f4"
server_key_iv: "1b279fc4591cc2168983259c44cc9da1"
server_key_version: "encryptionkey@0"
I20260927 21:12:15.405836 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:15.406548 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:15.406751 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:12:15.406842 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "5bef81cda84b486e9c1f941a408a0be5"
format_stamp: "Formatted at 2026-09-27 21:12:15 on dist-test-slave-r9x1"
server_key: "49c8e52a8c8a637f79f7b5d28121d9f4"
server_key_iv: "1b279fc4591cc2168983259c44cc9da1"
server_key_version: "encryptionkey@0"
I20260927 21:12:15.406924 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:15.421391 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:15.423512 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:15.423659 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:15.423887 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:15.424190 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:15.424223 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:15.424243 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:15.424259 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:15.429948 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43919
I20260927 21:12:15.429991 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43919 every 8 connection(s)
I20260927 21:12:15.430358 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:15.434239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 1292
I20260927 21:12:15.434337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:15.434484 392 external_mini_cluster.cc:1428] Setting key 63e2cf00a6a0495553dd9ff8ab0bf3de
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:15.440164 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.431184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33813 (local address 127.0.98.62:33241)
0927 21:12:15.431252 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.431254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.431437 (+ 183us) server_negotiation.cc:407] Connection header received
0927 21:12:15.431944 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.431945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.431955 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.432008 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:15.432961 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.433544 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.434828 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.435078 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.435151 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.437535 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:15.437536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:15.437537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:15.437544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:15.438793 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.439197 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.439198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.439198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.439248 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.439541 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.439542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.439543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.439688 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:15.439753 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.440016 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.440093 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:12:15.440337 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.431160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33241 (local address 127.0.98.3:33813)
0927 21:12:15.431315 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:15.431354 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:15.431847 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:15.432068 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:15.432071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:15.432655 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:15.432863 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.432867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.433636 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.433643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.434750 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.434753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.435163 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.435164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.435313 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.435844 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:15.435853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:15.437417 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:15.438883 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:15.438890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:15.438893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:15.439131 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:15.439304 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:15.439305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:15.439305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:15.439416 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:15.439760 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:15.439765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:15.439900 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 21:12:15.439990 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:12:15.440125 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260927 21:12:15.440760 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33241
I20260927 21:12:15.440850 1421 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:15.441018 1421 heartbeater.cc:511] Master 127.0.98.62:33241 requested a full tablet report, sending...
I20260927 21:12:15.441385 962 ts_manager.cc:194] Registered new tserver with Master: 5bef81cda84b486e9c1f941a408a0be5 (127.0.98.3:43919)
I20260927 21:12:15.441771 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33813
I20260927 21:12:15.446273 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:12:15.458756 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.447823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.0.98.62:33241)
0927 21:12:15.447914 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.447915 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.447964 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:12:15.448020 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.448020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.448029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.448072 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:15.448558 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.449063 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.449834 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.450039 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.450106 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.456210 (+ 6104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:15.456212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:15.456212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:15.456222 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:15.457603 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.457943 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.457944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.457945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.458000 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.458175 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.458175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.458176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.458370 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:15.458462 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.458577 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.458664 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:12:15.461122 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:15.462811 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:15.469560 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.465414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38271 (local address 127.0.0.1:59966)
0927 21:12:15.465713 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:15.465722 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:15.465776 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:15.466019 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:15.466020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:15.466022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:15.466077 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.466079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.467077 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.467078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.468326 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.468328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.469215 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.469216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.469331 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.469349 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 21:12:15.469414 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 21:12:15.469474 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":73,"threads_started":1}
I20260927 21:12:15.469660 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.465185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42987 (local address 127.0.0.1:42072)
0927 21:12:15.465362 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:15.465403 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:15.465468 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:15.466036 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:15.466037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:15.466038 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:15.466086 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.466088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.467404 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.467405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.468471 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.468473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.469365 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.469366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.469432 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.469435 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:15.469486 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:12:15.469532 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":7,"thread_start_us":63,"threads_started":1}
I20260927 21:12:15.469624 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.465336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.0.98.1:42987)
0927 21:12:15.465909 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.465913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.465922 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:15.465957 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.465959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.465972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.466040 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:15.466165 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.467283 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.468561 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.469304 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.469363 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.469415 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.469509 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.469547 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":141,"threads_started":1}
I20260927 21:12:15.470216 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.465575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.0.98.2:38271)
0927 21:12:15.465711 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.465716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.465768 (+ 52us) server_negotiation.cc:407] Connection header received
0927 21:12:15.465848 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.465850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.465874 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.465945 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:15.466181 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.466983 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.468398 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.469127 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.469204 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.469290 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.470086 (+ 796us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.470138 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260927 21:12:15.470413 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.465203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43919 (local address 127.0.0.1:48982)
0927 21:12:15.465711 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:15.465721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:15.465789 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:15.465957 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:15.465960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:15.465962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:15.466042 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.466044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.468535 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.468536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.469214 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.469215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.470151 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.470152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.470227 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.470231 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:15.470285 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:12:15.470324 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":104,"threads_started":1}
I20260927 21:12:15.470485 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 50c0152d397b4358a3f8e63117f8664f (DEFAULT_TABLE table=test-table [id=8dd7a0d9f76946dcabaaf0b0b80d27fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:15.470645 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.465554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.0.98.3:43919)
0927 21:12:15.465711 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.465716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.465771 (+ 55us) server_negotiation.cc:407] Connection header received
0927 21:12:15.465852 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.465854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.465869 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.465938 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:15.466139 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.467283 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.469334 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.470070 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.470142 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.470197 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.470526 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.470576 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260927 21:12:15.470882 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c0152d397b4358a3f8e63117f8664f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:15.470968 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 50c0152d397b4358a3f8e63117f8664f (DEFAULT_TABLE table=test-table [id=8dd7a0d9f76946dcabaaf0b0b80d27fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:15.471280 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c0152d397b4358a3f8e63117f8664f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:15.474086 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 50c0152d397b4358a3f8e63117f8664f (DEFAULT_TABLE table=test-table [id=8dd7a0d9f76946dcabaaf0b0b80d27fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:15.474511 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c0152d397b4358a3f8e63117f8664f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:15.477092 1439 tablet_bootstrap.cc:492] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b: Bootstrap starting.
I20260927 21:12:15.477240 1438 tablet_bootstrap.cc:492] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8: Bootstrap starting.
I20260927 21:12:15.477434 1440 tablet_bootstrap.cc:492] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5: Bootstrap starting.
I20260927 21:12:15.477801 1439 tablet_bootstrap.cc:654] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:15.477995 1438 tablet_bootstrap.cc:654] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:15.477995 1440 tablet_bootstrap.cc:654] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:15.478317 1439 log.cc:824] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:15.478518 1438 log.cc:824] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:15.478556 1440 log.cc:824] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:15.479113 1439 tablet_bootstrap.cc:492] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b: No bootstrap required, opened a new log
I20260927 21:12:15.479180 1439 ts_tablet_manager.cc:1397] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:12:15.479257 1438 tablet_bootstrap.cc:492] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8: No bootstrap required, opened a new log
I20260927 21:12:15.479265 1440 tablet_bootstrap.cc:492] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5: No bootstrap required, opened a new log
I20260927 21:12:15.479327 1440 ts_tablet_manager.cc:1397] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:15.479327 1438 ts_tablet_manager.cc:1397] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:12:15.481376 1438 raft_consensus.cc:348] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } }
I20260927 21:12:15.481606 1438 raft_consensus.cc:374] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:15.481642 1438 raft_consensus.cc:729] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8cea4078a744fe892e496d748181d8, State: Initialized, Role: FOLLOWER
I20260927 21:12:15.481642 1439 raft_consensus.cc:348] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } }
I20260927 21:12:15.481853 1439 raft_consensus.cc:374] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:15.481818 1438 consensus_queue.cc:260] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } }
I20260927 21:12:15.481896 1439 raft_consensus.cc:729] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84879e8a7111448f9085d4b9f209274b, State: Initialized, Role: FOLLOWER
I20260927 21:12:15.481801 1440 raft_consensus.cc:348] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } }
I20260927 21:12:15.481976 1440 raft_consensus.cc:374] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:15.482019 1440 raft_consensus.cc:729] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bef81cda84b486e9c1f941a408a0be5, State: Initialized, Role: FOLLOWER
I20260927 21:12:15.482036 1439 consensus_queue.cc:260] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } }
I20260927 21:12:15.482122 1438 ts_tablet_manager.cc:1428] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:12:15.482131 1440 consensus_queue.cc:260] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } }
I20260927 21:12:15.482254 1287 heartbeater.cc:503] Master 127.0.98.62:33241 was elected leader, sending a full tablet report...
I20260927 21:12:15.482314 1439 ts_tablet_manager.cc:1428] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:12:15.482383 1421 heartbeater.cc:503] Master 127.0.98.62:33241 was elected leader, sending a full tablet report...
I20260927 21:12:15.482393 1440 ts_tablet_manager.cc:1428] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:12:15.482504 1153 heartbeater.cc:503] Master 127.0.98.62:33241 was elected leader, sending a full tablet report...
W20260927 21:12:15.526912 1288 tablet.cc:2369] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:15.580405 1444 raft_consensus.cc:482] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:15.580533 1444 raft_consensus.cc:504] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } }
I20260927 21:12:15.580893 1444 leader_election.cc:288] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84879e8a7111448f9085d4b9f209274b (127.0.98.1:42987), 5bef81cda84b486e9c1f941a408a0be5 (127.0.98.3:43919)
I20260927 21:12:15.585305 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.581189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40675 (local address 127.0.98.1:42987)
0927 21:12:15.581234 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.581236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.581444 (+ 208us) server_negotiation.cc:407] Connection header received
0927 21:12:15.581475 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.581476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.581486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.581532 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:15.581656 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.582371 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.584043 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.584632 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.584801 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.584819 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.585141 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.585200 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:12:15.585410 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.581216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57865 (local address 127.0.98.3:43919)
0927 21:12:15.581283 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.581285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.581295 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:15.581323 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.581325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.581335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.581386 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:15.581563 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.582293 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.584004 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.584628 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.584907 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.584921 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.585256 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.585332 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:12:15.585414 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.581117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43919 (local address 127.0.98.2:57865)
0927 21:12:15.581165 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:15.581177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:15.581230 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:15.581419 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:15.581420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:15.581421 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:15.581492 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.581494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.582409 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.582410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.583863 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.583868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.584926 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.584927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.585017 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.585026 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:12:15.585127 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:12:15.585342 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":42}
I20260927 21:12:15.585753 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.581120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42987 (local address 127.0.98.2:40675)
0927 21:12:15.581368 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:15.581377 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:15.581434 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:15.581547 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:15.581547 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:15.581549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:15.581603 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.581604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.582456 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.582457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.583863 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:15.583868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.584876 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.584879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:15.585015 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.585029 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:12:15.585127 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:12:15.585411 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":291,"thread_start_us":136,"threads_started":1}
I20260927 21:12:15.585983 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c0152d397b4358a3f8e63117f8664f" candidate_uuid: "8f8cea4078a744fe892e496d748181d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bef81cda84b486e9c1f941a408a0be5" is_pre_election: true
I20260927 21:12:15.586166 1375 raft_consensus.cc:2454] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f8cea4078a744fe892e496d748181d8 in term 0.
I20260927 21:12:15.586259 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c0152d397b4358a3f8e63117f8664f" candidate_uuid: "8f8cea4078a744fe892e496d748181d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84879e8a7111448f9085d4b9f209274b" is_pre_election: true
I20260927 21:12:15.586429 1107 raft_consensus.cc:2454] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f8cea4078a744fe892e496d748181d8 in term 0.
I20260927 21:12:15.586427 1175 leader_election.cc:302] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bef81cda84b486e9c1f941a408a0be5, 8f8cea4078a744fe892e496d748181d8; no voters:
I20260927 21:12:15.586562 1444 raft_consensus.cc:2781] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:15.586660 1444 raft_consensus.cc:482] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:15.586685 1444 raft_consensus.cc:3037] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:15.587394 1444 raft_consensus.cc:504] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } }
I20260927 21:12:15.587517 1444 leader_election.cc:288] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [CANDIDATE]: Term 1 election: Requested vote from peers 84879e8a7111448f9085d4b9f209274b (127.0.98.1:42987), 5bef81cda84b486e9c1f941a408a0be5 (127.0.98.3:43919)
I20260927 21:12:15.587677 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c0152d397b4358a3f8e63117f8664f" candidate_uuid: "8f8cea4078a744fe892e496d748181d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84879e8a7111448f9085d4b9f209274b"
I20260927 21:12:15.587756 1107 raft_consensus.cc:3037] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:15.588474 1107 raft_consensus.cc:2454] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f8cea4078a744fe892e496d748181d8 in term 1.
I20260927 21:12:15.588591 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c0152d397b4358a3f8e63117f8664f" candidate_uuid: "8f8cea4078a744fe892e496d748181d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bef81cda84b486e9c1f941a408a0be5"
I20260927 21:12:15.588673 1176 leader_election.cc:302] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84879e8a7111448f9085d4b9f209274b, 8f8cea4078a744fe892e496d748181d8; no voters:
I20260927 21:12:15.588676 1375 raft_consensus.cc:3037] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:15.588778 1444 raft_consensus.cc:2781] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:15.588928 1444 raft_consensus.cc:686] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [term 1 LEADER]: Becoming Leader. State: Replica: 8f8cea4078a744fe892e496d748181d8, State: Running, Role: LEADER
I20260927 21:12:15.589031 1444 consensus_queue.cc:237] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } }
I20260927 21:12:15.589622 1375 raft_consensus.cc:2454] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f8cea4078a744fe892e496d748181d8 in term 1.
I20260927 21:12:15.589924 961 catalog_manager.cc:5495] T 50c0152d397b4358a3f8e63117f8664f P 8f8cea4078a744fe892e496d748181d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f8cea4078a744fe892e496d748181d8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8f8cea4078a744fe892e496d748181d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84879e8a7111448f9085d4b9f209274b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f8cea4078a744fe892e496d748181d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bef81cda84b486e9c1f941a408a0be5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43919 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260927 21:12:15.619632 1154 tablet.cc:2369] T 50c0152d397b4358a3f8e63117f8664f P 84879e8a7111448f9085d4b9f209274b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:15.622551 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.615212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.0.98.1:42987)
0927 21:12:15.615293 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.615295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.615306 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:12:15.615382 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.615383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.615394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.615484 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:15.616038 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.616517 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.617250 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.617523 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.617577 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.619994 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:15.620005 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:15.620005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:15.620013 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:15.621553 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.621938 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.621941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.621941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.621992 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.622140 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.622141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.622141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.622273 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:15.622387 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.622420 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.622483 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:12:15.646389 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.638830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.0.98.1:42987)
0927 21:12:15.638922 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.638923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.638936 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:12:15.639003 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.639004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.639014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.639056 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:15.639636 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.640137 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.640940 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.641140 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.641193 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.643485 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:15.643487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:15.643487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:15.643495 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:15.644798 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.645235 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.645237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.645237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.645424 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.645730 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.645732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.645733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.645895 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:15.646012 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.646203 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.646296 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:12:15.671046 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:15.664022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.0.98.1:42987)
0927 21:12:15.664076 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:15.664078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:15.664198 (+ 120us) server_negotiation.cc:407] Connection header received
0927 21:12:15.664274 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:15.664275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:15.664286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:15.664335 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:15.664894 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.665423 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.666197 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:15.666389 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:15.666442 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:15.668737 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:15.668739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:15.668739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:15.668748 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:15.670015 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.670347 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.670348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.670349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.670397 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:15.670584 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:15.670585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:15.670585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:15.670683 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:15.670766 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:15.670876 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 21:12:15.670951 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 21:12:15.671460 1067 tablet_service.cc:2908] Rejecting scan request for tablet 50c0152d397b4358a3f8e63117f8664f: Uninitialized: clean time has not yet been initialized
I20260927 21:12:15.672462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 1024
I20260927 21:12:15.677091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 1158
W20260927 21:12:15.680958 1422 tablet.cc:2369] T 50c0152d397b4358a3f8e63117f8664f P 5bef81cda84b486e9c1f941a408a0be5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:15.681681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 1292
I20260927 21:12:15.685616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 930
2026-09-27T21:12:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1963 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[1462](info): set up 2 sockets
Sep 27 21:12:15 dist-test-slave-r9x1 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 21:12:16.640722 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:16 dist-test-slave-r9x1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:16.760669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36653
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:16.842887 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:16.843122 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:16.843146 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:16.844362 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:16.844411 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:12:16.844424 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:16.844435 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:16.844447 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:16.844458 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:16.846570 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36653
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:16.846979 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:16.847204 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:16.852099 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:16.852162 1475 server_base.cc:1034] running on GCE node
W20260927 21:12:16.852110 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:16.852126 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:16.852524 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:16.852823 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:16.853999 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790543536853956 us; error 53 us; skew 500 ppm
Sep 27 21:12:16 dist-test-slave-r9x1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:16.856923 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:16.857437 1475 webserver.cc:460] Webserver started at http://127.0.98.62:42435/ using document root <none> and password file <none>
I20260927 21:12:16.857697 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:16.857750 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:16.857873 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:16.858948 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "9cc761002348474cbb1e1bb5a0889f5c"
format_stamp: "Formatted at 2026-09-27 21:12:16 on dist-test-slave-r9x1"
server_key: "61f2fa9d03c7cee6ce6d05736b155f8b"
server_key_iv: "ca987dd027e5da97e16747a9c14219e3"
server_key_version: "encryptionkey@0"
I20260927 21:12:16.859285 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "9cc761002348474cbb1e1bb5a0889f5c"
format_stamp: "Formatted at 2026-09-27 21:12:16 on dist-test-slave-r9x1"
server_key: "61f2fa9d03c7cee6ce6d05736b155f8b"
server_key_iv: "ca987dd027e5da97e16747a9c14219e3"
server_key_version: "encryptionkey@0"
I20260927 21:12:16.860584 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:16.861369 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:16.861558 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:16.861646 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "9cc761002348474cbb1e1bb5a0889f5c"
format_stamp: "Formatted at 2026-09-27 21:12:16 on dist-test-slave-r9x1"
server_key: "61f2fa9d03c7cee6ce6d05736b155f8b"
server_key_iv: "ca987dd027e5da97e16747a9c14219e3"
server_key_version: "encryptionkey@0"
I20260927 21:12:16.861716 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:16.895041 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:16.897732 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:16.897898 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:16.902370 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36653
I20260927 21:12:16.902421 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36653 every 8 connection(s)
I20260927 21:12:16.902776 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:16.903453 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:16.906028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 1475
I20260927 21:12:16.906113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:16.906149 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c: Bootstrap starting.
I20260927 21:12:16.906244 392 external_mini_cluster.cc:1428] Setting key 4bd8d0b729ede4cce4472f59413f75a1
I20260927 21:12:16.906804 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:16.907125 1553 log.cc:824] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:16.907914 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c: No bootstrap required, opened a new log
I20260927 21:12:16.910233 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc761002348474cbb1e1bb5a0889f5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36653 } }
I20260927 21:12:16.910398 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:16.910420 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc761002348474cbb1e1bb5a0889f5c, State: Initialized, Role: FOLLOWER
I20260927 21:12:16.910534 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc761002348474cbb1e1bb5a0889f5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36653 } }
I20260927 21:12:16.910609 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:16.910637 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:16.910694 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:16.911276 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc761002348474cbb1e1bb5a0889f5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36653 } }
I20260927 21:12:16.911391 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc761002348474cbb1e1bb5a0889f5c; no voters:
I20260927 21:12:16.911566 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:16.911729 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:16.911829 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:16.911903 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [term 1 LEADER]: Becoming Leader. State: Replica: 9cc761002348474cbb1e1bb5a0889f5c, State: Running, Role: LEADER
I20260927 21:12:16.912019 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc761002348474cbb1e1bb5a0889f5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36653 } }
I20260927 21:12:16.912577 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc761002348474cbb1e1bb5a0889f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc761002348474cbb1e1bb5a0889f5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36653 } } }
I20260927 21:12:16.912604 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc761002348474cbb1e1bb5a0889f5c. Latest consensus state: current_term: 1 leader_uuid: "9cc761002348474cbb1e1bb5a0889f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc761002348474cbb1e1bb5a0889f5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36653 } } }
I20260927 21:12:16.912671 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:16.912678 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:16.913012 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 21:12:16 dist-test-slave-r9x1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:16.913595 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:16.915316 1564 catalog_manager.cc:1347] Generated new cluster ID: c122a51ecf6f4c68b3ae275da8c53bc6
I20260927 21:12:16.915377 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:16.917590 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:16.906789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.0.98.62:36653)
0927 21:12:16.906971 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:16.906982 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:16.907017 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:12:16.907571 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:16.907583 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:16.907654 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:16.907815 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:16.908425 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:16.909860 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:16.910759 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:16.911203 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:16.911286 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:16.914159 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:16.914163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:16.914164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:16.914175 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:16.916084 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:16.916508 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:16.916512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:16.916513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:16.916607 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:16.916767 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:16.916768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:16.916768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:16.917019 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:16.917117 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:16.917223 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 21:12:16.917388 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260927 21:12:16.928414 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:16.929095 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:16.933193 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c: Generated new TSK 0
I20260927 21:12:16.933439 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:16.968209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36653
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:17.056349 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:17.056582 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:17.056609 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:17.057909 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:17.057957 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:17.057978 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:12:17.057996 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:17.059602 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:17.059818 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:17.060011 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:17.064913 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:17.064913 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:17.065092 1579 server_base.cc:1034] running on GCE node
W20260927 21:12:17.064917 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:17.065438 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:17.065719 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:17.066869 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790543537066844 us; error 39 us; skew 500 ppm
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:17.069581 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:17.070065 1579 webserver.cc:460] Webserver started at http://127.0.98.1:42259/ using document root <none> and password file <none>
I20260927 21:12:17.070310 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:17.070362 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:17.070488 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:17.071775 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "8ecd62a04eaa4d078cd422bb7582cade"
format_stamp: "Formatted at 2026-09-27 21:12:17 on dist-test-slave-r9x1"
server_key: "cd303fabbb21f9973b4a1ffe9c7c2082"
server_key_iv: "75b0b8325f223a357c08b38d1d48fb00"
server_key_version: "encryptionkey@0"
I20260927 21:12:17.072155 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8ecd62a04eaa4d078cd422bb7582cade"
format_stamp: "Formatted at 2026-09-27 21:12:17 on dist-test-slave-r9x1"
server_key: "cd303fabbb21f9973b4a1ffe9c7c2082"
server_key_iv: "75b0b8325f223a357c08b38d1d48fb00"
server_key_version: "encryptionkey@0"
I20260927 21:12:17.073690 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:17.074604 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.074875 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:17.074983 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "8ecd62a04eaa4d078cd422bb7582cade"
format_stamp: "Formatted at 2026-09-27 21:12:17 on dist-test-slave-r9x1"
server_key: "cd303fabbb21f9973b4a1ffe9c7c2082"
server_key_iv: "75b0b8325f223a357c08b38d1d48fb00"
server_key_version: "encryptionkey@0"
I20260927 21:12:17.075057 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:17.089895 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:17.092136 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:17.092283 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:17.092875 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:17.092989 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.093037 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:17.093055 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:17.100056 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42009
I20260927 21:12:17.100155 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42009 every 8 connection(s)
I20260927 21:12:17.100492 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:12:17.103238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 1579
I20260927 21:12:17.103325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:17.103430 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.093342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34111 (local address 127.0.98.62:36653)
0927 21:12:17.093412 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.093414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.093656 (+ 242us) server_negotiation.cc:407] Connection header received
0927 21:12:17.094156 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.094158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.094172 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.094247 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.094931 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.095678 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.097367 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.097716 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.097797 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.100509 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.100512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.100512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.100521 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.101945 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.102404 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.102405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.102405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.102460 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.102727 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.102727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.102728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.102831 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.102878 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.103225 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.103334 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:12:17.103487 392 external_mini_cluster.cc:1428] Setting key e71a1581910bd3bd116035d4b6560aa8
I20260927 21:12:17.103567 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.093356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36653 (local address 127.0.98.1:34111)
0927 21:12:17.093522 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.093563 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.094048 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.094312 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.094317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.094608 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:17.094832 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.094838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.095790 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.095791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.097239 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.097244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.097839 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.097840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.098007 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.098593 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:17.098604 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:17.100394 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:17.102030 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.102037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.102039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.102325 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.102543 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.102544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.102545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.102668 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.102905 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:17.102911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:17.103045 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.103170 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.103340 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260927 21:12:17.104040 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36653
I20260927 21:12:17.104132 1710 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:17.104311 1710 heartbeater.cc:511] Master 127.0.98.62:36653 requested a full tablet report, sending...
I20260927 21:12:17.104857 1507 ts_manager.cc:194] Registered new tserver with Master: 8ecd62a04eaa4d078cd422bb7582cade (127.0.98.1:42009)
I20260927 21:12:17.105803 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34111
I20260927 21:12:17.111593 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.105252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38053 (local address 127.0.98.62:36653)
0927 21:12:17.105307 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.105310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.105487 (+ 177us) server_negotiation.cc:407] Connection header received
0927 21:12:17.105521 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.105522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.105532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.105591 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.105963 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.106682 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.107720 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.107925 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.107982 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.109069 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.109070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.109071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.109077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.110408 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.110881 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.110882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.110882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.110941 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.111138 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.111138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.111139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.111262 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.111307 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.111454 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.111522 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:12:17.111675 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.105193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36653 (local address 127.0.98.1:38053)
0927 21:12:17.105408 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.105419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.105481 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.105595 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.105596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.105802 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:17.105873 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.105876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.106798 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.106799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.107615 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.107618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.107997 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.107997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.108059 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.108542 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:17.108548 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:17.108954 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:17.110499 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.110500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.110500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.110733 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.111011 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.111012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.111012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.111078 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.111304 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:17.111305 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:17.111363 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.111439 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.111550 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":164,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:17.132267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36653
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:17.214599 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:17.214839 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:17.214865 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:17.216089 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:17.216133 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:17.216154 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:12:17.216171 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:17.217826 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:17.218043 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:17.218235 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:17.223366 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:17.223357 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:17.223357 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:17.223816 1719 server_base.cc:1034] running on GCE node
I20260927 21:12:17.224042 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:17.224284 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:17.225425 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790543537225406 us; error 29 us; skew 500 ppm
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:17.227943 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:17.228403 1719 webserver.cc:460] Webserver started at http://127.0.98.2:38851/ using document root <none> and password file <none>
I20260927 21:12:17.228621 1719 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:17.228669 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:17.228797 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:17.229951 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "44322b20cb2e421c86c79bdfd83d0bc9"
format_stamp: "Formatted at 2026-09-27 21:12:17 on dist-test-slave-r9x1"
server_key: "20970a3ebd2f1dc1ae1fdee56cb652e7"
server_key_iv: "3868d681739ce3a9784ce18d6cdf7131"
server_key_version: "encryptionkey@0"
I20260927 21:12:17.230293 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44322b20cb2e421c86c79bdfd83d0bc9"
format_stamp: "Formatted at 2026-09-27 21:12:17 on dist-test-slave-r9x1"
server_key: "20970a3ebd2f1dc1ae1fdee56cb652e7"
server_key_iv: "3868d681739ce3a9784ce18d6cdf7131"
server_key_version: "encryptionkey@0"
I20260927 21:12:17.231673 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:17.232475 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.232694 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:17.232798 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "44322b20cb2e421c86c79bdfd83d0bc9"
format_stamp: "Formatted at 2026-09-27 21:12:17 on dist-test-slave-r9x1"
server_key: "20970a3ebd2f1dc1ae1fdee56cb652e7"
server_key_iv: "3868d681739ce3a9784ce18d6cdf7131"
server_key_version: "encryptionkey@0"
I20260927 21:12:17.232867 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:17.245433 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:17.247610 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:17.247754 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:17.248260 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:17.248301 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.248333 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:17.248358 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.255290 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32801
I20260927 21:12:17.255367 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32801 every 8 connection(s)
I20260927 21:12:17.255724 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:17.257158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 1719
I20260927 21:12:17.257246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:17.257452 392 external_mini_cluster.cc:1428] Setting key 0abd2014970537eb8435f4cf469c78cd
I20260927 21:12:17.259296 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.248792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37511 (local address 127.0.98.62:36653)
0927 21:12:17.248848 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.248850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.249231 (+ 381us) server_negotiation.cc:407] Connection header received
0927 21:12:17.249873 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.249874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.249888 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.249946 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.250849 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.251621 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.253181 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.253462 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.253521 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.256397 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.256398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.256399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.256406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.257914 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.258390 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.258391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.258391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.258451 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.258710 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.258711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.258711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.258817 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.258863 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.259162 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.259232 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:12:17.259558 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.248883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36653 (local address 127.0.98.2:37511)
0927 21:12:17.249025 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.249060 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.249761 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.250053 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.250059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.250430 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:17.250743 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.250750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.251719 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.251720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.253091 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.253095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.253541 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.253542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.253709 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.254558 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:17.254571 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:17.256289 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:17.257987 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.257995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.257997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.258314 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.258513 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.258514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.258514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.258652 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.258866 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:17.258871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:17.259035 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.259142 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.259322 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260927 21:12:17.260093 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36653
I20260927 21:12:17.260183 1850 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:17.260433 1850 heartbeater.cc:511] Master 127.0.98.62:36653 requested a full tablet report, sending...
I20260927 21:12:17.260818 1507 ts_manager.cc:194] Registered new tserver with Master: 44322b20cb2e421c86c79bdfd83d0bc9 (127.0.98.2:32801)
I20260927 21:12:17.261339 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37511
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:12:17.266820 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.261101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54025 (local address 127.0.98.62:36653)
0927 21:12:17.261139 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.261140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.261249 (+ 109us) server_negotiation.cc:407] Connection header received
0927 21:12:17.261351 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.261352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.261362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.261392 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.261831 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.262421 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.263095 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.263294 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.263351 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.264280 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.264281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.264281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.264288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.265653 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.266070 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.266071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.266071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.266135 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.266303 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.266303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.266304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.266448 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.266507 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.266644 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.266724 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:12:17.266834 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.261057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36653 (local address 127.0.98.2:54025)
0927 21:12:17.261184 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.261193 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.261248 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.261451 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.261452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.261674 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:17.261743 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.261745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.262513 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.262513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.263018 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.263019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.263360 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.263361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.263426 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.263881 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:17.263886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:17.264174 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:17.265733 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.265734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.265735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.265984 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.266191 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.266192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.266193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.266253 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.266506 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:17.266507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:17.266554 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.266620 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.266722 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:17.283869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36653
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:17.365584 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:17.365814 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:17.365839 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:17.367103 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:17.367156 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:17.367178 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:12:17.367210 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:17.368885 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:17.369140 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:17.369398 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:17.374457 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:17.374495 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:17.374547 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:17.374773 1859 server_base.cc:1034] running on GCE node
I20260927 21:12:17.375003 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:17.375260 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:17.376428 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790543537376406 us; error 35 us; skew 500 ppm
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:17.379475 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:17.379972 1859 webserver.cc:460] Webserver started at http://127.0.98.3:35025/ using document root <none> and password file <none>
I20260927 21:12:17.380196 1859 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:17.380249 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:17.380378 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:17.381603 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "75eb4632951541fc8b2a6467b11bba89"
format_stamp: "Formatted at 2026-09-27 21:12:17 on dist-test-slave-r9x1"
server_key: "c97bcf856bd39c963bc14aa59b66a45b"
server_key_iv: "3e6153be96d7d82f74b29ffbb948e909"
server_key_version: "encryptionkey@0"
I20260927 21:12:17.381964 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "75eb4632951541fc8b2a6467b11bba89"
format_stamp: "Formatted at 2026-09-27 21:12:17 on dist-test-slave-r9x1"
server_key: "c97bcf856bd39c963bc14aa59b66a45b"
server_key_iv: "3e6153be96d7d82f74b29ffbb948e909"
server_key_version: "encryptionkey@0"
I20260927 21:12:17.383646 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:17.384558 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.384747 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:17.384845 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "75eb4632951541fc8b2a6467b11bba89"
format_stamp: "Formatted at 2026-09-27 21:12:17 on dist-test-slave-r9x1"
server_key: "c97bcf856bd39c963bc14aa59b66a45b"
server_key_iv: "3e6153be96d7d82f74b29ffbb948e909"
server_key_version: "encryptionkey@0"
I20260927 21:12:17.384917 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:17.397758 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:17.400090 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:17.400238 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:17.400748 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:17.400790 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.400820 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:17.400837 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.407186 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41365
I20260927 21:12:17.407280 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41365 every 8 connection(s)
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:17.407719 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:17.408764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 1859
I20260927 21:12:17.408849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:17.409026 392 external_mini_cluster.cc:1428] Setting key e351e5af41f9b6bc11eb608fb14c8e71
I20260927 21:12:17.410888 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.401197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50205 (local address 127.0.98.62:36653)
0927 21:12:17.401285 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.401288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.401455 (+ 167us) server_negotiation.cc:407] Connection header received
0927 21:12:17.401936 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.401938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.401951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.402025 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.402781 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.403647 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.405249 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.405559 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.405660 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.408160 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.408162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.408162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.408170 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.409591 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.410001 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.410002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.410003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.410058 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.410284 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.410285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.410285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.410403 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.410457 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.410719 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.410800 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:12:17.411068 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.401157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36653 (local address 127.0.98.3:50205)
0927 21:12:17.401328 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.401369 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.401847 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.402085 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.402089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.402398 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:17.402680 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.402686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.403761 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.403762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.405150 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.405154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.405674 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.405675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.405788 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.406314 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:17.406325 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:17.408037 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:17.409667 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.409674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.409677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.409938 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.410117 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.410118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.410118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.410229 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.410454 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:17.410460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:17.410598 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.410690 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.410854 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260927 21:12:17.411520 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36653
I20260927 21:12:17.411662 1990 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:17.411847 1990 heartbeater.cc:511] Master 127.0.98.62:36653 requested a full tablet report, sending...
I20260927 21:12:17.412254 1505 ts_manager.cc:194] Registered new tserver with Master: 75eb4632951541fc8b2a6467b11bba89 (127.0.98.3:41365)
I20260927 21:12:17.412715 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50205
I20260927 21:12:17.418251 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.412570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36653 (local address 127.0.98.3:34053)
0927 21:12:17.412714 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.412726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.412791 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.413038 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.413039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.413218 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:17.413295 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.413297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.413979 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.413980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.414846 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.414849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.415218 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.415218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.415287 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.415735 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:17.415740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:17.416034 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:17.417305 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.417306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.417307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.417466 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.417631 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.417631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.417632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.417715 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.417932 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:17.417933 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:17.417976 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.418036 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.418138 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260927 21:12:17.418243 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.412666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34053 (local address 127.0.98.62:36653)
0927 21:12:17.412709 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.412711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.412803 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:12:17.412971 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.412972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.412983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.413035 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.413359 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.413898 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.414940 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.415151 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.415207 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.416127 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.416128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.416129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.416136 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.417205 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.417527 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.417528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.417528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.417581 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.417768 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.417769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.417770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.417881 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.417934 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.418063 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.418137 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:12:17.420758 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543537, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543537, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:17.452427 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.445573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.0.98.62:36653)
0927 21:12:17.445640 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.445642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.445654 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:12:17.445686 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.445687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.445697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.445780 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.446450 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.446945 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.447690 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.447887 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.447943 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.450284 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.450286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.450286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.450295 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.451374 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.451723 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.451725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.451725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.451781 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.451937 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.451938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.451938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.452057 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.452144 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.452253 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.452336 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:12:17.454228 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:17.455740 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:17.461937 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.457898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32801 (local address 127.0.0.1:55540)
0927 21:12:17.458230 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.458278 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.458332 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.458524 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.458527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.458529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:17.458585 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.458587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.459546 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.459547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.460611 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.460614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.461637 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.461638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.461720 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.461732 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.461794 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.461851 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":79,"threads_started":1}
I20260927 21:12:17.461956 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.458019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.0.98.2:32801)
0927 21:12:17.458374 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.458379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.458389 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:17.458427 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.458429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.458445 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.458521 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:17.458655 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.459460 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.460705 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.461457 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.461514 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.461594 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.461828 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.461866 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":84,"threads_started":1}
I20260927 21:12:17.462606 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.458070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41365 (local address 127.0.0.1:58516)
0927 21:12:17.458697 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.458707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.458759 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.458974 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.458975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.458978 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:17.459015 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.459016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.460246 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.460247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.461218 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.461220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.462403 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.462404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.462473 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.462476 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.462519 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.462551 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":75,"threads_started":1}
I20260927 21:12:17.462703 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.458179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.0.98.3:41365)
0927 21:12:17.458352 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.458357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.458757 (+ 400us) server_negotiation.cc:407] Connection header received
0927 21:12:17.458827 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.458829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.458845 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.458912 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:17.459083 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.460192 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.461322 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.461980 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.462290 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.462375 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.462553 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.462612 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260927 21:12:17.462816 1784 tablet_service.cc:1467] Processing CreateTablet for tablet a65b9b0f6a3342dba93e59faecf83721 (DEFAULT_TABLE table=test-table [id=3ae2686212f743adaf03b8129a264a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:17.463330 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65b9b0f6a3342dba93e59faecf83721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:17.463711 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a65b9b0f6a3342dba93e59faecf83721 (DEFAULT_TABLE table=test-table [id=3ae2686212f743adaf03b8129a264a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:17.464017 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65b9b0f6a3342dba93e59faecf83721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:17.466480 2013 tablet_bootstrap.cc:492] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: Bootstrap starting.
I20260927 21:12:17.466815 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.457872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42009 (local address 127.0.0.1:52126)
0927 21:12:17.458230 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.458262 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.458330 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.458623 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.458623 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.458625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:17.458682 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.458683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.460186 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.460187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.461437 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.461440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.462310 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.462311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.462371 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.462374 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.466662 (+ 4288us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.466733 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":27,"thread_start_us":56,"threads_started":1}
I20260927 21:12:17.466903 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.458055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.0.98.1:42009)
0927 21:12:17.458481 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.458484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.458493 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:17.458525 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.458527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.458540 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.458610 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:17.458925 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.460097 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.461541 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.462231 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.462304 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.462374 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.466741 (+ 4367us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.466801 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":76,"threads_started":1}
I20260927 21:12:17.467195 2013 tablet_bootstrap.cc:654] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:17.467494 2014 tablet_bootstrap.cc:492] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: Bootstrap starting.
I20260927 21:12:17.467561 2013 log.cc:824] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:17.467823 1644 tablet_service.cc:1467] Processing CreateTablet for tablet a65b9b0f6a3342dba93e59faecf83721 (DEFAULT_TABLE table=test-table [id=3ae2686212f743adaf03b8129a264a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:17.468094 2014 tablet_bootstrap.cc:654] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:17.468155 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65b9b0f6a3342dba93e59faecf83721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:17.468391 2013 tablet_bootstrap.cc:492] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: No bootstrap required, opened a new log
I20260927 21:12:17.468405 2014 log.cc:824] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:17.468451 2013 ts_tablet_manager.cc:1397] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:17.469173 2014 tablet_bootstrap.cc:492] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: No bootstrap required, opened a new log
I20260927 21:12:17.469249 2014 ts_tablet_manager.cc:1397] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:17.470489 2013 raft_consensus.cc:348] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } }
I20260927 21:12:17.470707 2013 raft_consensus.cc:374] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:17.470777 2013 raft_consensus.cc:729] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44322b20cb2e421c86c79bdfd83d0bc9, State: Initialized, Role: FOLLOWER
I20260927 21:12:17.470929 2013 consensus_queue.cc:260] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } }
I20260927 21:12:17.471225 2013 ts_tablet_manager.cc:1428] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:12:17.471253 1850 heartbeater.cc:503] Master 127.0.98.62:36653 was elected leader, sending a full tablet report...
I20260927 21:12:17.471244 2014 raft_consensus.cc:348] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } }
I20260927 21:12:17.471383 2014 raft_consensus.cc:374] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:17.471414 2014 raft_consensus.cc:729] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75eb4632951541fc8b2a6467b11bba89, State: Initialized, Role: FOLLOWER
I20260927 21:12:17.471408 2018 tablet_bootstrap.cc:492] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: Bootstrap starting.
I20260927 21:12:17.471521 2014 consensus_queue.cc:260] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } }
I20260927 21:12:17.471767 2014 ts_tablet_manager.cc:1428] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:17.471823 1990 heartbeater.cc:503] Master 127.0.98.62:36653 was elected leader, sending a full tablet report...
I20260927 21:12:17.472043 2018 tablet_bootstrap.cc:654] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:17.472373 2018 log.cc:824] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:17.473228 2018 tablet_bootstrap.cc:492] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: No bootstrap required, opened a new log
I20260927 21:12:17.473349 2018 ts_tablet_manager.cc:1397] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:17.474879 2018 raft_consensus.cc:348] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } }
I20260927 21:12:17.475019 2018 raft_consensus.cc:374] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:17.475041 2018 raft_consensus.cc:729] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ecd62a04eaa4d078cd422bb7582cade, State: Initialized, Role: FOLLOWER
I20260927 21:12:17.475147 2018 consensus_queue.cc:260] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } }
I20260927 21:12:17.475411 2018 ts_tablet_manager.cc:1428] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:17.475473 1710 heartbeater.cc:503] Master 127.0.98.62:36653 was elected leader, sending a full tablet report...
W20260927 21:12:17.506305 1851 tablet.cc:2369] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:17.548444 2017 raft_consensus.cc:482] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:17.548585 2017 raft_consensus.cc:504] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } }
I20260927 21:12:17.548970 2017 leader_election.cc:288] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ecd62a04eaa4d078cd422bb7582cade (127.0.98.1:42009), 75eb4632951541fc8b2a6467b11bba89 (127.0.98.3:41365)
I20260927 21:12:17.552230 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.549507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34929 (local address 127.0.98.3:41365)
0927 21:12:17.549547 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.549549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.549683 (+ 134us) server_negotiation.cc:407] Connection header received
0927 21:12:17.549709 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.549710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.549720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.549772 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:17.550063 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.550573 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.551517 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.551894 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.551960 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.551971 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.552118 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.552157 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:12:17.552253 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.549358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38249 (local address 127.0.98.1:42009)
0927 21:12:17.549424 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.549426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.549438 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:12:17.549475 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.549477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.549487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.549547 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:17.549745 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.550358 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.551470 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.551883 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.551942 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.551953 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.552122 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.552182 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:12:17.552289 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.549394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41365 (local address 127.0.98.2:34929)
0927 21:12:17.549547 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.549555 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.549602 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.549816 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.549817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.549819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:17.549948 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.549951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.550673 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.550674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.551429 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.551432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.551975 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.551976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.552051 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.552055 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.552111 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.552153 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260927 21:12:17.552296 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.549230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42009 (local address 127.0.98.2:38249)
0927 21:12:17.549332 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.549347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.549421 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.549602 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.549603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.549605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:17.549667 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.549669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.550437 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.550438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.551369 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.551372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.551962 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.551963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.552055 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.552059 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.552107 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.552154 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":37}
I20260927 21:12:17.552767 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65b9b0f6a3342dba93e59faecf83721" candidate_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75eb4632951541fc8b2a6467b11bba89" is_pre_election: true
I20260927 21:12:17.552788 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65b9b0f6a3342dba93e59faecf83721" candidate_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ecd62a04eaa4d078cd422bb7582cade" is_pre_election: true
I20260927 21:12:17.552920 1944 raft_consensus.cc:2454] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44322b20cb2e421c86c79bdfd83d0bc9 in term 0.
I20260927 21:12:17.552925 1664 raft_consensus.cc:2454] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44322b20cb2e421c86c79bdfd83d0bc9 in term 0.
I20260927 21:12:17.553165 1736 leader_election.cc:302] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44322b20cb2e421c86c79bdfd83d0bc9, 8ecd62a04eaa4d078cd422bb7582cade; no voters:
I20260927 21:12:17.553366 2017 raft_consensus.cc:2781] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:17.553443 2017 raft_consensus.cc:482] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:17.553462 2017 raft_consensus.cc:3037] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:17.554447 2017 raft_consensus.cc:504] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } }
I20260927 21:12:17.554581 2017 leader_election.cc:288] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 8ecd62a04eaa4d078cd422bb7582cade (127.0.98.1:42009), 75eb4632951541fc8b2a6467b11bba89 (127.0.98.3:41365)
I20260927 21:12:17.554791 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65b9b0f6a3342dba93e59faecf83721" candidate_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ecd62a04eaa4d078cd422bb7582cade"
I20260927 21:12:17.554845 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65b9b0f6a3342dba93e59faecf83721" candidate_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75eb4632951541fc8b2a6467b11bba89"
I20260927 21:12:17.554883 1664 raft_consensus.cc:3037] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:17.554930 1944 raft_consensus.cc:3037] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:17.555600 1944 raft_consensus.cc:2454] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44322b20cb2e421c86c79bdfd83d0bc9 in term 1.
I20260927 21:12:17.555610 1664 raft_consensus.cc:2454] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44322b20cb2e421c86c79bdfd83d0bc9 in term 1.
I20260927 21:12:17.555794 1736 leader_election.cc:302] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44322b20cb2e421c86c79bdfd83d0bc9, 75eb4632951541fc8b2a6467b11bba89; no voters:
I20260927 21:12:17.555892 2017 raft_consensus.cc:2781] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:17.556028 2017 raft_consensus.cc:686] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 44322b20cb2e421c86c79bdfd83d0bc9, State: Running, Role: LEADER
I20260927 21:12:17.556111 2017 consensus_queue.cc:237] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } }
I20260927 21:12:17.556963 1505 catalog_manager.cc:5495] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44322b20cb2e421c86c79bdfd83d0bc9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } health_report { overall_health: HEALTHY } } }
I20260927 21:12:17.583454 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.580873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.0.98.2:32801)
0927 21:12:17.580945 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.580948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.580956 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:17.580995 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.580996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.581018 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.581075 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:17.581221 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.581745 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.582534 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.582732 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.582787 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.583109 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:17.583223 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:17.583298 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.583347 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.583371 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:12:17.585439 1664 raft_consensus.cc:1261] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 FOLLOWER]: Refusing update from remote peer 44322b20cb2e421c86c79bdfd83d0bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:17.585459 1944 raft_consensus.cc:1261] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Refusing update from remote peer 44322b20cb2e421c86c79bdfd83d0bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:17.585781 2023 consensus_queue.cc:1035] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:17.585939 2017 consensus_queue.cc:1035] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:17.588878 2026 mvcc.cc:205] Tried to move back new op lower bound from 7334066329946726400 to 7334066329830305792. Current Snapshot: MvccSnapshot[applied={T|T < 7334066329946726400}]
I20260927 21:12:17.590312 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334066329946726400 to 7334066329830305792. Current Snapshot: MvccSnapshot[applied={T|T < 7334066329946726400}]
I20260927 21:12:17.592567 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:17.592664 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:17.593513 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c: Sending DeleteTablet for 3 replicas of tablet a65b9b0f6a3342dba93e59faecf83721
I20260927 21:12:17.594213 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet a65b9b0f6a3342dba93e59faecf83721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55540
I20260927 21:12:17.594152 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet a65b9b0f6a3342dba93e59faecf83721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52126
I20260927 21:12:17.594218 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet a65b9b0f6a3342dba93e59faecf83721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58516
I20260927 21:12:17.594467 2035 tablet_replica.cc:330] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: stopping tablet replica
I20260927 21:12:17.594565 2035 raft_consensus.cc:2229] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:17.594779 2037 tablet_replica.cc:330] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: stopping tablet replica
I20260927 21:12:17.594743 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:17.594914 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:17.594952 2037 raft_consensus.cc:2229] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:17.594970 2035 raft_consensus.cc:2258] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:17.595052 2037 raft_consensus.cc:2258] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:17.595544 2037 ts_tablet_manager.cc:1905] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:17.595870 2035 ts_tablet_manager.cc:1905] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:17.596383 1924 tablet_service.cc:1467] Processing CreateTablet for tablet ee424df633b640fe9cbae9bc859e2004 (DEFAULT_TABLE table=test-table [id=87ad30694b6349c0a8771fc63dd7de39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:17.596489 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee424df633b640fe9cbae9bc859e2004. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:17.597245 2037 ts_tablet_manager.cc:1918] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:17.597352 2037 log.cc:1196] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/a65b9b0f6a3342dba93e59faecf83721
I20260927 21:12:17.597525 2035 ts_tablet_manager.cc:1918] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:17.597589 2035 log.cc:1196] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/a65b9b0f6a3342dba93e59faecf83721
I20260927 21:12:17.597591 2037 ts_tablet_manager.cc:1939] T a65b9b0f6a3342dba93e59faecf83721 P 44322b20cb2e421c86c79bdfd83d0bc9: Deleting consensus metadata
I20260927 21:12:17.597811 2036 tablet_replica.cc:330] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: stopping tablet replica
I20260927 21:12:17.597841 2035 ts_tablet_manager.cc:1939] T a65b9b0f6a3342dba93e59faecf83721 P 75eb4632951541fc8b2a6467b11bba89: Deleting consensus metadata
I20260927 21:12:17.597939 2036 raft_consensus.cc:2229] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:17.597952 1784 tablet_service.cc:1467] Processing CreateTablet for tablet ee424df633b640fe9cbae9bc859e2004 (DEFAULT_TABLE table=test-table [id=87ad30694b6349c0a8771fc63dd7de39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:17.598047 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee424df633b640fe9cbae9bc859e2004. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:17.598048 2036 raft_consensus.cc:2258] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:17.598580 1491 catalog_manager.cc:4841] TS 44322b20cb2e421c86c79bdfd83d0bc9 (127.0.98.2:32801): tablet a65b9b0f6a3342dba93e59faecf83721 (table test-table [id=3ae2686212f743adaf03b8129a264a27]) successfully deleted
I20260927 21:12:17.598904 2036 ts_tablet_manager.cc:1905] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:17.599228 2013 tablet_bootstrap.cc:492] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: Bootstrap starting.
I20260927 21:12:17.599262 2014 tablet_bootstrap.cc:492] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: Bootstrap starting.
I20260927 21:12:17.599634 2013 tablet_bootstrap.cc:654] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:17.599643 2014 tablet_bootstrap.cc:654] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:17.600390 1492 catalog_manager.cc:4841] TS 75eb4632951541fc8b2a6467b11bba89 (127.0.98.3:41365): tablet a65b9b0f6a3342dba93e59faecf83721 (table test-table [id=3ae2686212f743adaf03b8129a264a27]) successfully deleted
I20260927 21:12:17.600427 2013 tablet_bootstrap.cc:492] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: No bootstrap required, opened a new log
I20260927 21:12:17.600468 2013 ts_tablet_manager.cc:1397] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:17.600610 2013 raft_consensus.cc:348] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.600692 2013 raft_consensus.cc:374] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:17.600713 2013 raft_consensus.cc:729] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44322b20cb2e421c86c79bdfd83d0bc9, State: Initialized, Role: FOLLOWER
I20260927 21:12:17.600765 2013 consensus_queue.cc:260] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.600844 2013 ts_tablet_manager.cc:1428] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:17.601552 1644 tablet_service.cc:1467] Processing CreateTablet for tablet ee424df633b640fe9cbae9bc859e2004 (DEFAULT_TABLE table=test-table [id=87ad30694b6349c0a8771fc63dd7de39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:17.601704 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee424df633b640fe9cbae9bc859e2004. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:17.601929 2036 ts_tablet_manager.cc:1918] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:17.601999 2036 log.cc:1196] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/a65b9b0f6a3342dba93e59faecf83721
I20260927 21:12:17.602248 2036 ts_tablet_manager.cc:1939] T a65b9b0f6a3342dba93e59faecf83721 P 8ecd62a04eaa4d078cd422bb7582cade: Deleting consensus metadata
I20260927 21:12:17.602890 2018 tablet_bootstrap.cc:492] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: Bootstrap starting.
I20260927 21:12:17.602916 1492 catalog_manager.cc:4841] TS 8ecd62a04eaa4d078cd422bb7582cade (127.0.98.1:42009): tablet a65b9b0f6a3342dba93e59faecf83721 (table test-table [id=3ae2686212f743adaf03b8129a264a27]) successfully deleted
I20260927 21:12:17.603242 2014 tablet_bootstrap.cc:492] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: No bootstrap required, opened a new log
I20260927 21:12:17.603281 2014 ts_tablet_manager.cc:1397] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 21:12:17.603405 2014 raft_consensus.cc:348] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.603461 2014 raft_consensus.cc:374] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:17.603472 2014 raft_consensus.cc:729] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75eb4632951541fc8b2a6467b11bba89, State: Initialized, Role: FOLLOWER
I20260927 21:12:17.603508 2014 consensus_queue.cc:260] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.603588 2014 ts_tablet_manager.cc:1428] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.603809 2018 tablet_bootstrap.cc:654] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:17.604415 2018 tablet_bootstrap.cc:492] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: No bootstrap required, opened a new log
I20260927 21:12:17.604465 2018 ts_tablet_manager.cc:1397] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:17.604590 2018 raft_consensus.cc:348] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.604655 2018 raft_consensus.cc:374] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:17.604673 2018 raft_consensus.cc:729] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ecd62a04eaa4d078cd422bb7582cade, State: Initialized, Role: FOLLOWER
I20260927 21:12:17.604718 2018 consensus_queue.cc:260] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.604799 2018 ts_tablet_manager.cc:1428] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260927 21:12:17.658375 1991 tablet.cc:2369] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:17.700943 2021 raft_consensus.cc:482] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:17.701066 2021 raft_consensus.cc:504] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.701452 2021 leader_election.cc:288] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44322b20cb2e421c86c79bdfd83d0bc9 (127.0.98.2:32801), 75eb4632951541fc8b2a6467b11bba89 (127.0.98.3:41365)
I20260927 21:12:17.704679 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.701799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39687 (local address 127.0.98.2:32801)
0927 21:12:17.701847 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.701848 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.701858 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:17.701913 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.701914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.701923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.701970 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:17.702148 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.702894 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.703897 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.704353 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.704426 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.704438 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.704567 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.704602 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:12:17.704695 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.701639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32801 (local address 127.0.98.1:39687)
0927 21:12:17.701784 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.701796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.701851 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.701987 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.701988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.701990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:17.702069 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.702071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.703007 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.703008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.703804 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.703806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.704431 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.704431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.704497 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.704502 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.704555 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.704601 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260927 21:12:17.705019 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee424df633b640fe9cbae9bc859e2004" candidate_uuid: "8ecd62a04eaa4d078cd422bb7582cade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" is_pre_election: true
I20260927 21:12:17.705156 1804 raft_consensus.cc:2454] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ecd62a04eaa4d078cd422bb7582cade in term 0.
I20260927 21:12:17.705289 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.701765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47571 (local address 127.0.98.3:41365)
0927 21:12:17.701811 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.701813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.701943 (+ 130us) server_negotiation.cc:407] Connection header received
0927 21:12:17.702041 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.702043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.702052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.702104 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:17.702251 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.702945 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.704358 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.704855 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.704920 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.704929 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.705169 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.705208 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:12:17.705260 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.701639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41365 (local address 127.0.98.1:47571)
0927 21:12:17.701868 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.701878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.701956 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.702107 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.702107 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.702117 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:17.702184 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.702186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.703049 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.703050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.704231 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.704233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.704960 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.704961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.705084 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.705088 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.705133 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.705178 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":2,"thread_start_us":42,"threads_started":1}
I20260927 21:12:17.705422 1595 leader_election.cc:302] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44322b20cb2e421c86c79bdfd83d0bc9, 8ecd62a04eaa4d078cd422bb7582cade; no voters:
I20260927 21:12:17.705477 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee424df633b640fe9cbae9bc859e2004" candidate_uuid: "8ecd62a04eaa4d078cd422bb7582cade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75eb4632951541fc8b2a6467b11bba89" is_pre_election: true
I20260927 21:12:17.705542 2021 raft_consensus.cc:2781] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:17.705551 1944 raft_consensus.cc:2454] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ecd62a04eaa4d078cd422bb7582cade in term 0.
I20260927 21:12:17.705567 2021 raft_consensus.cc:482] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:17.705586 2021 raft_consensus.cc:3037] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:17.706424 2021 raft_consensus.cc:504] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.706576 2021 leader_election.cc:288] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [CANDIDATE]: Term 1 election: Requested vote from peers 44322b20cb2e421c86c79bdfd83d0bc9 (127.0.98.2:32801), 75eb4632951541fc8b2a6467b11bba89 (127.0.98.3:41365)
I20260927 21:12:17.706751 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee424df633b640fe9cbae9bc859e2004" candidate_uuid: "8ecd62a04eaa4d078cd422bb7582cade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44322b20cb2e421c86c79bdfd83d0bc9"
I20260927 21:12:17.706769 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee424df633b640fe9cbae9bc859e2004" candidate_uuid: "8ecd62a04eaa4d078cd422bb7582cade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75eb4632951541fc8b2a6467b11bba89"
I20260927 21:12:17.706838 1944 raft_consensus.cc:3037] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:17.706838 1804 raft_consensus.cc:3037] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:17.707599 1804 raft_consensus.cc:2454] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ecd62a04eaa4d078cd422bb7582cade in term 1.
I20260927 21:12:17.707599 1944 raft_consensus.cc:2454] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ecd62a04eaa4d078cd422bb7582cade in term 1.
I20260927 21:12:17.707793 1596 leader_election.cc:302] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75eb4632951541fc8b2a6467b11bba89, 8ecd62a04eaa4d078cd422bb7582cade; no voters:
I20260927 21:12:17.707909 2021 raft_consensus.cc:2781] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:17.708015 2021 raft_consensus.cc:686] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 LEADER]: Becoming Leader. State: Replica: 8ecd62a04eaa4d078cd422bb7582cade, State: Running, Role: LEADER
I20260927 21:12:17.708087 2021 consensus_queue.cc:237] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.708774 1505 catalog_manager.cc:5495] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ecd62a04eaa4d078cd422bb7582cade (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8ecd62a04eaa4d078cd422bb7582cade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:17.760798 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.754084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36653 (local address 127.0.0.1:37260)
0927 21:12:17.754236 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.754247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.754317 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.754426 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.754427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.754734 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:17.754784 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.754786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.755469 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.755470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.756004 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.756006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.756362 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.756362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.756400 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.756943 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:17.756948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:17.758382 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:17.759800 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.759807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.759809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.760020 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.760224 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:17.760225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:17.760225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:17.760275 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:17.760475 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:17.760479 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:17.760522 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.760571 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.760659 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260927 21:12:17.760895 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.754127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.0.98.62:36653)
0927 21:12:17.754188 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.754190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.754313 (+ 123us) server_negotiation.cc:407] Connection header received
0927 21:12:17.754361 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.754362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.754371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.754419 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.754884 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.755390 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.756099 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.756297 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.756354 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.758502 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.758504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.758504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.758511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.759714 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.760109 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.760111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.760111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.760167 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.760319 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.760319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.760320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.760429 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.760480 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.760637 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.760702 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:12:17.763011 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:12:17.763204 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:17.764499 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 96033369771c46d79760090d163104b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1c3a18026294fccb62954a1a6b27fe1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:12:17.764519 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 96033369771c46d79760090d163104b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1c3a18026294fccb62954a1a6b27fe1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:12:17.764665 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96033369771c46d79760090d163104b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:17.764667 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96033369771c46d79760090d163104b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:17.764597 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 96033369771c46d79760090d163104b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1c3a18026294fccb62954a1a6b27fe1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:12:17.764772 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96033369771c46d79760090d163104b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:17.766003 2014 tablet_bootstrap.cc:492] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: Bootstrap starting.
I20260927 21:12:17.766206 2013 tablet_bootstrap.cc:492] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: Bootstrap starting.
I20260927 21:12:17.766412 2014 tablet_bootstrap.cc:654] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:17.766522 2013 tablet_bootstrap.cc:654] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:17.766775 2018 tablet_bootstrap.cc:492] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: Bootstrap starting.
I20260927 21:12:17.767149 2014 tablet_bootstrap.cc:492] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: No bootstrap required, opened a new log
I20260927 21:12:17.767184 2018 tablet_bootstrap.cc:654] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:17.767218 2014 ts_tablet_manager.cc:1397] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:17.767371 2014 raft_consensus.cc:348] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.767454 2014 raft_consensus.cc:374] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:17.767482 2014 raft_consensus.cc:729] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75eb4632951541fc8b2a6467b11bba89, State: Initialized, Role: FOLLOWER
I20260927 21:12:17.767526 2014 consensus_queue.cc:260] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.767640 2014 ts_tablet_manager.cc:1428] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:17.767647 2013 tablet_bootstrap.cc:492] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: No bootstrap required, opened a new log
I20260927 21:12:17.767696 2013 ts_tablet_manager.cc:1397] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:17.767781 2019 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.767827 2013 raft_consensus.cc:348] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.767866 2019 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:17.767895 2013 raft_consensus.cc:374] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:17.767915 2013 raft_consensus.cc:729] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44322b20cb2e421c86c79bdfd83d0bc9, State: Initialized, Role: FOLLOWER
I20260927 21:12:17.767958 2013 consensus_queue.cc:260] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.768042 2013 ts_tablet_manager.cc:1428] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.768061 2017 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.768160 2017 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:17.768258 2018 tablet_bootstrap.cc:492] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: No bootstrap required, opened a new log
I20260927 21:12:17.768299 2018 ts_tablet_manager.cc:1397] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:17.768450 2018 raft_consensus.cc:348] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.768513 2018 raft_consensus.cc:374] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:17.768530 2018 raft_consensus.cc:729] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ecd62a04eaa4d078cd422bb7582cade, State: Initialized, Role: FOLLOWER
I20260927 21:12:17.768565 2018 consensus_queue.cc:260] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.768648 2018 ts_tablet_manager.cc:1428] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:17.768702 2021 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.768780 2021 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 21:12:17.851402 1711 tablet.cc:2369] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:17.875398 2019 raft_consensus.cc:482] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:17.875512 2019 raft_consensus.cc:504] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.875869 2019 leader_election.cc:288] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ecd62a04eaa4d078cd422bb7582cade (127.0.98.1:42009), 44322b20cb2e421c86c79bdfd83d0bc9 (127.0.98.2:32801)
I20260927 21:12:17.879405 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.876078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42009 (local address 127.0.98.3:47151)
0927 21:12:17.876162 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.876175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.876237 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.876389 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.876389 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.876391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:17.876460 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.876462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.877420 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.877422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.878586 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.878590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.879176 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.879177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.879248 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.879252 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.879300 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.879338 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 21:12:17.879432 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.876190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47151 (local address 127.0.98.1:42009)
0927 21:12:17.876237 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.876239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.876252 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:12:17.876286 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.876287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.876294 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.876377 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:17.876530 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.877341 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.878695 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.879077 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.879145 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.879155 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.879314 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.879353 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:12:17.879714 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96033369771c46d79760090d163104b3" candidate_uuid: "75eb4632951541fc8b2a6467b11bba89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ecd62a04eaa4d078cd422bb7582cade" is_pre_election: true
I20260927 21:12:17.879809 1664 raft_consensus.cc:2454] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75eb4632951541fc8b2a6467b11bba89 in term 0.
I20260927 21:12:17.879992 1876 leader_election.cc:302] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75eb4632951541fc8b2a6467b11bba89, 8ecd62a04eaa4d078cd422bb7582cade; no voters:
I20260927 21:12:17.880182 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.876190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37713 (local address 127.0.98.2:32801)
0927 21:12:17.876237 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.876239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.876330 (+ 91us) server_negotiation.cc:407] Connection header received
0927 21:12:17.876371 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.876372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.876384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.876432 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:17.876567 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.877317 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.879267 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.879824 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.879897 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.879907 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.880070 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.880122 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:12:17.880153 2019 raft_consensus.cc:2781] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:17.880167 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.876089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32801 (local address 127.0.98.3:37713)
0927 21:12:17.876265 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.876273 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.876324 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.876446 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.876447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.876448 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:17.876507 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.876508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.877420 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.877422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.879170 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.879172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.879913 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.879914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.879997 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.880001 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.880044 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.880079 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":15,"thread_start_us":101,"threads_started":1}
I20260927 21:12:17.880221 2019 raft_consensus.cc:482] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:17.880244 2019 raft_consensus.cc:3037] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:17.880414 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96033369771c46d79760090d163104b3" candidate_uuid: "75eb4632951541fc8b2a6467b11bba89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" is_pre_election: true
I20260927 21:12:17.880512 1804 raft_consensus.cc:2454] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75eb4632951541fc8b2a6467b11bba89 in term 0.
I20260927 21:12:17.880986 2019 raft_consensus.cc:504] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.881129 2019 leader_election.cc:288] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [CANDIDATE]: Term 1 election: Requested vote from peers 8ecd62a04eaa4d078cd422bb7582cade (127.0.98.1:42009), 44322b20cb2e421c86c79bdfd83d0bc9 (127.0.98.2:32801)
I20260927 21:12:17.881306 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96033369771c46d79760090d163104b3" candidate_uuid: "75eb4632951541fc8b2a6467b11bba89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44322b20cb2e421c86c79bdfd83d0bc9"
I20260927 21:12:17.881388 1804 raft_consensus.cc:3037] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:17.881397 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96033369771c46d79760090d163104b3" candidate_uuid: "75eb4632951541fc8b2a6467b11bba89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ecd62a04eaa4d078cd422bb7582cade"
I20260927 21:12:17.881475 1664 raft_consensus.cc:3037] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:17.882215 1664 raft_consensus.cc:2454] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75eb4632951541fc8b2a6467b11bba89 in term 1.
I20260927 21:12:17.882215 1804 raft_consensus.cc:2454] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75eb4632951541fc8b2a6467b11bba89 in term 1.
I20260927 21:12:17.882391 1875 leader_election.cc:302] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44322b20cb2e421c86c79bdfd83d0bc9, 75eb4632951541fc8b2a6467b11bba89; no voters:
I20260927 21:12:17.882493 2019 raft_consensus.cc:2781] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:17.882617 2019 raft_consensus.cc:686] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [term 1 LEADER]: Becoming Leader. State: Replica: 75eb4632951541fc8b2a6467b11bba89, State: Running, Role: LEADER
I20260927 21:12:17.882678 2019 consensus_queue.cc:237] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } }
I20260927 21:12:17.883030 2055 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: TxnStatusTablet state changed. Reason: New leader 75eb4632951541fc8b2a6467b11bba89. Latest consensus state: current_term: 1 leader_uuid: "75eb4632951541fc8b2a6467b11bba89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.883136 2055 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:12:17.883363 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:12:17.883476 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:12:17.883484 1506 catalog_manager.cc:5495] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75eb4632951541fc8b2a6467b11bba89 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "75eb4632951541fc8b2a6467b11bba89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } health_report { overall_health: HEALTHY } } }
I20260927 21:12:17.915910 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.913030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.0.98.3:41365)
0927 21:12:17.913078 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.913080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.913089 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:17.913200 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.913201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.913218 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.913307 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:17.913454 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.914025 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.915137 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.915329 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.915399 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.915596 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:17.915702 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:17.915755 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.915827 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.915846 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:12:17.915974 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.912966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41365 (local address 127.0.0.1:58518)
0927 21:12:17.913044 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.913055 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.913098 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.913337 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.913338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.913340 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:17.913398 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.913400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.914127 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.914128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.915018 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.915023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.915416 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.915417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.915488 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.915513 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:17.915762 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:17.915769 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.915824 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.915885 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 21:12:17.917176 1664 raft_consensus.cc:1261] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 FOLLOWER]: Refusing update from remote peer 75eb4632951541fc8b2a6467b11bba89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:17.917248 1804 raft_consensus.cc:1261] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 FOLLOWER]: Refusing update from remote peer 75eb4632951541fc8b2a6467b11bba89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:17.917466 2055 consensus_queue.cc:1035] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:17.917584 2019 consensus_queue.cc:1035] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:17.918210 2021 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: TxnStatusTablet state changed. Reason: New leader 75eb4632951541fc8b2a6467b11bba89. Latest consensus state: current_term: 1 leader_uuid: "75eb4632951541fc8b2a6467b11bba89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.918215 2017 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: TxnStatusTablet state changed. Reason: New leader 75eb4632951541fc8b2a6467b11bba89. Latest consensus state: current_term: 1 leader_uuid: "75eb4632951541fc8b2a6467b11bba89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.918309 2017 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:17.918309 2021 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:17.919324 2019 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75eb4632951541fc8b2a6467b11bba89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.919414 2019 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:12:17.919461 2055 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75eb4632951541fc8b2a6467b11bba89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.919519 2055 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 75eb4632951541fc8b2a6467b11bba89: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:12:17.919802 2017 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75eb4632951541fc8b2a6467b11bba89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.919901 2017 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 44322b20cb2e421c86c79bdfd83d0bc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:17.921382 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334066331306717184 to 7334066331167784960. Current Snapshot: MvccSnapshot[applied={T|T < 7334066331306717184 or (T in {7334066331306717184})}]
I20260927 21:12:17.921499 2021 tablet_replica.cc:439] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75eb4632951541fc8b2a6467b11bba89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd62a04eaa4d078cd422bb7582cade" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42009 } } peers { permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 } } }
I20260927 21:12:17.921609 2021 tablet_replica.cc:442] T 96033369771c46d79760090d163104b3 P 8ecd62a04eaa4d078cd422bb7582cade: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:17.924196 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.921252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.0.98.1:42009)
0927 21:12:17.921322 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.921324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.921332 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:17.921409 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.921410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.921433 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.921494 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:17.921666 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.922401 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.923249 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.923566 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.923653 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.923754 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:17.923910 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:17.923979 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.924088 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.924115 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:12:17.929597 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.926965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41261 (local address 127.0.98.3:41365)
0927 21:12:17.927009 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.927011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.927081 (+ 70us) server_negotiation.cc:407] Connection header received
0927 21:12:17.927149 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.927150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.927158 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.927202 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:17.927335 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.927796 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.928899 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.929083 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.929139 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.929359 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:17.929399 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:17.929453 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.929516 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.929536 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:12:17.929672 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.926861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41365 (local address 127.0.98.1:41261)
0927 21:12:17.927023 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.927035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.927098 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.927207 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.927208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.927210 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:17.927281 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.927283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.927865 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.927866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.928818 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.928821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.929168 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.929169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.929234 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.929254 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:17.929463 (+ 209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:17.929465 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.929512 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.929559 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260927 21:12:17.932220 1944 raft_consensus.cc:1261] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Refusing update from remote peer 8ecd62a04eaa4d078cd422bb7582cade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:17.932229 1804 raft_consensus.cc:1261] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 FOLLOWER]: Refusing update from remote peer 8ecd62a04eaa4d078cd422bb7582cade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:17.932511 2043 consensus_queue.cc:1035] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [LEADER]: Connected to new peer: Peer: permanent_uuid: "44322b20cb2e421c86c79bdfd83d0bc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:17.932664 2021 consensus_queue.cc:1035] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [LEADER]: Connected to new peer: Peer: permanent_uuid: "75eb4632951541fc8b2a6467b11bba89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:17.941418 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.938826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42009 (local address 127.0.98.3:48625)
0927 21:12:17.938997 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:17.939008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:17.939053 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:17.939221 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:17.939222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:17.939224 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:17.939279 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.939281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.940054 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.940055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.940667 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:17.940669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.940999 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.941000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:17.941053 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.941068 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:17.941220 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:17.941221 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:12:17.941261 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 21:12:17.941341 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260927 21:12:17.941395 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.939070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48625 (local address 127.0.98.1:42009)
0927 21:12:17.939111 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.939112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.939122 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:17.939143 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.939143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.939153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.939204 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:17.939362 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.939983 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.940742 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.940942 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.940995 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.941134 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:17.941169 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:17.941212 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.941295 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.941315 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:12:17.952668 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:17.952798 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:17.953487 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cc761002348474cbb1e1bb5a0889f5c: Sending DeleteTablet for 3 replicas of tablet ee424df633b640fe9cbae9bc859e2004
I20260927 21:12:17.953758 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet ee424df633b640fe9cbae9bc859e2004 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52126
I20260927 21:12:17.953824 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet ee424df633b640fe9cbae9bc859e2004 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55540
I20260927 21:12:17.953850 2036 tablet_replica.cc:330] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: stopping tablet replica
I20260927 21:12:17.953896 2036 raft_consensus.cc:2229] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:17.953902 2037 tablet_replica.cc:330] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: stopping tablet replica
I20260927 21:12:17.953974 2037 raft_consensus.cc:2229] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:17.953982 2036 raft_consensus.cc:2258] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:17.954010 2037 raft_consensus.cc:2258] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:17.954200 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet ee424df633b640fe9cbae9bc859e2004 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58516
I20260927 21:12:17.954295 2035 tablet_replica.cc:330] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: stopping tablet replica
I20260927 21:12:17.954312 2036 ts_tablet_manager.cc:1905] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:17.954334 2035 raft_consensus.cc:2229] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:17.954366 2035 raft_consensus.cc:2258] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:17.954663 2035 ts_tablet_manager.cc:1905] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:17.956228 2035 ts_tablet_manager.cc:1918] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:17.956270 2036 ts_tablet_manager.cc:1918] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:17.956300 2035 log.cc:1196] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/ee424df633b640fe9cbae9bc859e2004
I20260927 21:12:17.956315 2036 log.cc:1196] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/ee424df633b640fe9cbae9bc859e2004
I20260927 21:12:17.956496 2036 ts_tablet_manager.cc:1939] T ee424df633b640fe9cbae9bc859e2004 P 8ecd62a04eaa4d078cd422bb7582cade: Deleting consensus metadata
I20260927 21:12:17.956496 2035 ts_tablet_manager.cc:1939] T ee424df633b640fe9cbae9bc859e2004 P 75eb4632951541fc8b2a6467b11bba89: Deleting consensus metadata
I20260927 21:12:17.957139 1492 catalog_manager.cc:4841] TS 8ecd62a04eaa4d078cd422bb7582cade (127.0.98.1:42009): tablet ee424df633b640fe9cbae9bc859e2004 (table test-table [id=87ad30694b6349c0a8771fc63dd7de39]) successfully deleted
I20260927 21:12:17.957580 1492 catalog_manager.cc:4841] TS 75eb4632951541fc8b2a6467b11bba89 (127.0.98.3:41365): tablet ee424df633b640fe9cbae9bc859e2004 (table test-table [id=87ad30694b6349c0a8771fc63dd7de39]) successfully deleted
I20260927 21:12:17.957685 2037 ts_tablet_manager.cc:1905] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:17.959030 2037 ts_tablet_manager.cc:1918] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:17.959090 2037 log.cc:1196] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/ee424df633b640fe9cbae9bc859e2004
I20260927 21:12:17.959286 2037 ts_tablet_manager.cc:1939] T ee424df633b640fe9cbae9bc859e2004 P 44322b20cb2e421c86c79bdfd83d0bc9: Deleting consensus metadata
I20260927 21:12:17.959630 1491 catalog_manager.cc:4841] TS 44322b20cb2e421c86c79bdfd83d0bc9 (127.0.98.2:32801): tablet ee424df633b640fe9cbae9bc859e2004 (table test-table [id=87ad30694b6349c0a8771fc63dd7de39]) successfully deleted
Sep 27 21:12:17 dist-test-slave-r9x1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543537, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:12:17.967715 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.959530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.0.98.1:42009)
0927 21:12:17.959569 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.959571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.959579 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:17.959602 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.959604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.959613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.959662 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.960342 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.960841 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.961650 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.961856 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.961910 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.964942 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.964951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.964952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.964962 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.966583 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.966927 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.966931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.966931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.966984 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.967150 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.967151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.967151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.967296 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.967450 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.967496 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.967592 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:12:17.967996 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52142
I20260927 21:12:17.974543 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:17.968939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.0.98.62:36653)
0927 21:12:17.968983 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:17.968985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:17.969017 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:12:17.969093 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:17.969094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:17.969104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:17.969149 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:17.969748 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.970240 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.970903 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:17.971095 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:17.971149 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:17.972264 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:17.972266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:17.972266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:17.972273 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:17.973549 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.973856 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.973857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.973858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.973911 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:17.974073 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:17.974074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:17.974074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:17.974187 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:17.974283 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:17.974370 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 21:12:17.974449 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 21:12:17.974776 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37268
I20260927 21:12:17.975437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 1579
I20260927 21:12:17.980947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 1719
I20260927 21:12:17.989132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 1859
I20260927 21:12:17.993898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 1475
2026-09-27T21:12:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2308 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:18 dist-test-slave-r9x1 krb5kdc[2079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:18 dist-test-slave-r9x1 krb5kdc[2079](info): set up 2 sockets
Sep 27 21:12:18 dist-test-slave-r9x1 krb5kdc[2079](info): commencing operation
krb5kdc: starting...
W20260927 21:12:18.960876 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:19.076411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36737
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:19.157236 2092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:19.157513 2092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:19.157546 2092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:19.158850 2092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:19.158906 2092 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:12:19.158922 2092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:19.158939 2092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:19.158954 2092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:19.158970 2092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:19.160630 2092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36737
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:19.160921 2092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:19.161154 2092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:19.165987 2098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:19.166119 2092 server_base.cc:1034] running on GCE node
W20260927 21:12:19.165980 2097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:19.166057 2100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:19.166512 2092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:19.166792 2092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:19.167956 2092 hybrid_clock.cc:648] HybridClock initialized: now 1790543539167934 us; error 31 us; skew 500 ppm
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:19.170608 2092 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:19.171075 2092 webserver.cc:460] Webserver started at http://127.0.98.62:45041/ using document root <none> and password file <none>
I20260927 21:12:19.171286 2092 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:19.171329 2092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:19.171429 2092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:19.172482 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "3aae6856c04e4e98b89dd9858edf794f"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "dba6cd95786e7dcdc8a5000f3bc94c7c"
server_key_iv: "79c8072da7b871d8db6fa9d6c98d3c9f"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.172775 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "3aae6856c04e4e98b89dd9858edf794f"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "dba6cd95786e7dcdc8a5000f3bc94c7c"
server_key_iv: "79c8072da7b871d8db6fa9d6c98d3c9f"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.174100 2092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:19.174798 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.174953 2092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:19.175034 2092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "3aae6856c04e4e98b89dd9858edf794f"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "dba6cd95786e7dcdc8a5000f3bc94c7c"
server_key_iv: "79c8072da7b871d8db6fa9d6c98d3c9f"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.175105 2092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:19.184077 2092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:19.186259 2092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:19.186390 2092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:19.190961 2092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36737
I20260927 21:12:19.191008 2169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36737 every 8 connection(s)
I20260927 21:12:19.191339 2092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:19.191944 2170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:19.194231 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f: Bootstrap starting.
I20260927 21:12:19.194814 2170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:19.195073 2170 log.cc:824] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:19.195749 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f: No bootstrap required, opened a new log
I20260927 21:12:19.197926 2170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aae6856c04e4e98b89dd9858edf794f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36737 } }
I20260927 21:12:19.198086 2170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:19.198107 2170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aae6856c04e4e98b89dd9858edf794f, State: Initialized, Role: FOLLOWER
I20260927 21:12:19.198242 2170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aae6856c04e4e98b89dd9858edf794f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36737 } }
I20260927 21:12:19.198316 2170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:19.198342 2170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:19.198390 2170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:19.198963 2170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aae6856c04e4e98b89dd9858edf794f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36737 } }
I20260927 21:12:19.199060 2170 leader_election.cc:302] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aae6856c04e4e98b89dd9858edf794f; no voters:
I20260927 21:12:19.199231 2170 leader_election.cc:288] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:19.199272 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:19.199433 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [term 1 LEADER]: Becoming Leader. State: Replica: 3aae6856c04e4e98b89dd9858edf794f, State: Running, Role: LEADER
I20260927 21:12:19.199504 2170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:19.199532 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aae6856c04e4e98b89dd9858edf794f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36737 } }
I20260927 21:12:19.199900 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aae6856c04e4e98b89dd9858edf794f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aae6856c04e4e98b89dd9858edf794f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36737 } } }
I20260927 21:12:19.199914 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aae6856c04e4e98b89dd9858edf794f. Latest consensus state: current_term: 1 leader_uuid: "3aae6856c04e4e98b89dd9858edf794f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aae6856c04e4e98b89dd9858edf794f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36737 } } }
I20260927 21:12:19.200088 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:19.200246 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:19.200486 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:19.200996 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:19.201408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 2092
I20260927 21:12:19.201498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:19.201651 392 external_mini_cluster.cc:1428] Setting key f18ce7bf524457e7e28f2a2511e36656
I20260927 21:12:19.202643 2182 catalog_manager.cc:1347] Generated new cluster ID: 7203f833f47b4081867d4f3106b3ba6b
I20260927 21:12:19.202746 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:19.214612 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.202181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.0.98.62:36737)
0927 21:12:19.202347 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.202357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.202388 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:12:19.202484 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.202490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.202533 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.202696 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:19.203301 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.204245 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.204997 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.205513 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.205614 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.211394 (+ 5780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:19.211399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:19.211399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:19.211413 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:19.212964 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.213382 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.213386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.213386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.213479 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.213656 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.213657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.213658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.213938 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:19.214076 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.214189 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.214395 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260927 21:12:19.236706 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:19.237253 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:19.240042 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f: Generated new TSK 0
I20260927 21:12:19.240214 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:19.273545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36737
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:19.356699 2196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:19.356935 2196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:19.356961 2196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:19.358273 2196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:19.358322 2196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:19.358343 2196 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:12:19.358354 2196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:12:19.358373 2196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:19.359910 2196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:19.360146 2196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:19.360378 2196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:19.365302 2204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:19.365500 2196 server_base.cc:1034] running on GCE node
W20260927 21:12:19.365329 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:19.365329 2201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:19.365816 2196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:19.366094 2196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:19.367273 2196 hybrid_clock.cc:648] HybridClock initialized: now 1790543539367254 us; error 38 us; skew 500 ppm
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:19.370064 2196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:19.370554 2196 webserver.cc:460] Webserver started at http://127.0.98.1:38177/ using document root <none> and password file <none>
I20260927 21:12:19.370782 2196 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:19.370831 2196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:19.370944 2196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:19.372023 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "33a81ac2daf34a24875d81e326f5b897"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "d4c339fb9cb9e2df7f9a5f6aa2cd8f78"
server_key_iv: "094062be22a6ad53f6d591be0cc64f59"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.372352 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "33a81ac2daf34a24875d81e326f5b897"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "d4c339fb9cb9e2df7f9a5f6aa2cd8f78"
server_key_iv: "094062be22a6ad53f6d591be0cc64f59"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.373679 2196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:19.374451 2211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.374632 2196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:19.374723 2196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "33a81ac2daf34a24875d81e326f5b897"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "d4c339fb9cb9e2df7f9a5f6aa2cd8f78"
server_key_iv: "094062be22a6ad53f6d591be0cc64f59"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.374789 2196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:19.393715 2196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:19.395722 2196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:19.395872 2196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:19.396395 2196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:19.396435 2196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.396472 2196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:19.396492 2196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.403659 2196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37973
I20260927 21:12:19.403751 2326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37973 every 8 connection(s)
I20260927 21:12:19.404058 2196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:19.407795 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.396993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46533 (local address 127.0.98.62:36737)
0927 21:12:19.397049 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.397051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.397313 (+ 262us) server_negotiation.cc:407] Connection header received
0927 21:12:19.397891 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.397892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.397909 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.397960 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:19.398678 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.399551 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.401152 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.401498 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.401568 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.404765 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:19.404767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:19.404767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:19.404780 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:19.406219 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.406690 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.406691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.406692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.406780 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.407134 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.407135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.407136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.407281 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:19.407346 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.407610 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.407710 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:12:19.407931 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.396969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36737 (local address 127.0.98.1:46533)
0927 21:12:19.397158 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.397212 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.397774 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.398041 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.398046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.398369 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:19.398579 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.398585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.399664 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.399665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.401065 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.401069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.401585 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.401587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.401726 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.402593 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:19.402605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:19.404625 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:19.406333 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.406342 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.406344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.406593 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.406887 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.406888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.406888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.407033 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.407343 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:19.407347 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:19.407482 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.407567 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.407724 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260927 21:12:19.408483 2327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36737
I20260927 21:12:19.408605 2327 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:19.408612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 2196
I20260927 21:12:19.408689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:19.408835 2327 heartbeater.cc:511] Master 127.0.98.62:36737 requested a full tablet report, sending...
I20260927 21:12:19.408886 392 external_mini_cluster.cc:1428] Setting key fee913d1b693c8f555b0754088e7a552
I20260927 21:12:19.409401 2124 ts_manager.cc:194] Registered new tserver with Master: 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973)
I20260927 21:12:19.410192 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46533
I20260927 21:12:19.415839 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.409802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43851 (local address 127.0.98.62:36737)
0927 21:12:19.409851 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.409854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.410007 (+ 153us) server_negotiation.cc:407] Connection header received
0927 21:12:19.410065 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.410066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.410077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.410128 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:19.410501 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.411196 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.412023 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.412230 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.412290 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.413384 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:19.413386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:19.413386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:19.413395 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:19.414760 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.415116 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.415117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.415118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.415172 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.415340 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.415340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.415341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.415455 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:19.415512 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.415650 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.415739 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:12:19.415853 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.409698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36737 (local address 127.0.98.1:43851)
0927 21:12:19.409943 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.409952 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.410008 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.410151 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.410152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.410343 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:19.410430 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.410432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.411290 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.411291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.411924 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.411927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.412293 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.412293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.412371 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.412855 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:19.412860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:19.413261 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:19.414864 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.414866 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.414866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.415048 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.415228 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.415228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.415229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.415290 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.415509 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:19.415510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:19.415560 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.415621 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.415741 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:19.435629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36737
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:19.519768 2336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:19.519991 2336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:19.520025 2336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:19.521311 2336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:19.521364 2336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:19.521394 2336 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:12:19.521409 2336 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:12:19.521432 2336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:19.523046 2336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:19.523331 2336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:19.523567 2336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:19.528651 2341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:19.528681 2336 server_base.cc:1034] running on GCE node
W20260927 21:12:19.528651 2344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:19.528687 2342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:19.529091 2336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:19.529403 2336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:19.530582 2336 hybrid_clock.cc:648] HybridClock initialized: now 1790543539530567 us; error 39 us; skew 500 ppm
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:19.533190 2336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:19.533700 2336 webserver.cc:460] Webserver started at http://127.0.98.2:33447/ using document root <none> and password file <none>
I20260927 21:12:19.533921 2336 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:19.533963 2336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:19.534057 2336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:19.535096 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "76fe9bb937804f46b0f931e1a3a9d5a5"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "c374801b81faaf5edd9afe9a5330c3a1"
server_key_iv: "544798d54326fa2e82e05f0ef8a0bbe2"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.535404 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "76fe9bb937804f46b0f931e1a3a9d5a5"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "c374801b81faaf5edd9afe9a5330c3a1"
server_key_iv: "544798d54326fa2e82e05f0ef8a0bbe2"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.536677 2336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:19.537599 2351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.537792 2336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:19.537914 2336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "76fe9bb937804f46b0f931e1a3a9d5a5"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "c374801b81faaf5edd9afe9a5330c3a1"
server_key_iv: "544798d54326fa2e82e05f0ef8a0bbe2"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.537993 2336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:19.551394 2336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:19.553699 2336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:19.553855 2336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:19.554476 2336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:19.554535 2336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.554569 2336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:19.554581 2336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:19.561208 2336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39101
I20260927 21:12:19.561426 2466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39101 every 8 connection(s)
I20260927 21:12:19.561648 2336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:12:19.564822 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.554888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41943 (local address 127.0.98.62:36737)
0927 21:12:19.554950 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.554953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.555204 (+ 251us) server_negotiation.cc:407] Connection header received
0927 21:12:19.555703 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.555704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.555718 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.555808 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:19.556683 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.557460 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.558510 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.558705 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.558762 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.561301 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:19.561303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:19.561303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:19.561314 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:19.562982 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.563453 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.563455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.563455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.563519 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.563873 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.563875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.563876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.564147 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:19.564258 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.564628 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.564722 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:12:19.564908 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.554887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36737 (local address 127.0.98.2:41943)
0927 21:12:19.555072 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.555116 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.555611 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.555898 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.555902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.556267 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:19.556586 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.556594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.557555 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.557556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.558415 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.558418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.558791 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.558792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.558914 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.559442 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:19.559450 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:19.561145 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:19.563094 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.563101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.563103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.563343 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.563602 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.563603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.563603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.563715 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.564296 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:19.564302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:19.564447 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.564556 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.564701 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260927 21:12:19.565553 2467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36737
I20260927 21:12:19.565644 2467 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:19.565894 2467 heartbeater.cc:511] Master 127.0.98.62:36737 requested a full tablet report, sending...
I20260927 21:12:19.566315 2124 ts_manager.cc:194] Registered new tserver with Master: 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101)
I20260927 21:12:19.566820 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41943
I20260927 21:12:19.570439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 2336
I20260927 21:12:19.570533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:19.570695 392 external_mini_cluster.cc:1428] Setting key e95eaa31abd08574f7b0d4b0791ae98b
I20260927 21:12:19.573334 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.566802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52991 (local address 127.0.98.62:36737)
0927 21:12:19.566874 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.566876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.567104 (+ 228us) server_negotiation.cc:407] Connection header received
0927 21:12:19.567205 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.567206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.567219 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.567277 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:19.567729 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.568270 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.569057 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.569260 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.569348 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.570446 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:19.570448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:19.570449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:19.570458 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:19.572063 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.572467 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.572468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.572469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.572536 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.572716 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.572717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.572717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.572885 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:19.572947 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.573099 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.573191 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:12:19.573346 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.566621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36737 (local address 127.0.98.2:52991)
0927 21:12:19.566960 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.566976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.567132 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.567284 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.567285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.567558 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:19.567632 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.567636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.568378 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.568380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.568963 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.568965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.569366 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.569368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.569436 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.569977 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:19.569982 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:19.570353 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:19.572175 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.572177 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.572177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.572393 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.572593 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.572594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.572594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.572656 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.572949 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:19.572949 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:19.573001 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.573067 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.573203 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:19.596917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36737
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:19.679154 2476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:19.679384 2476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:19.679430 2476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:19.680678 2476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:19.680729 2476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:19.680756 2476 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:12:19.680770 2476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:12:19.680794 2476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:19.682451 2476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:19.682715 2476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:19.682938 2476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:12:19.688301 2476 server_base.cc:1034] running on GCE node
W20260927 21:12:19.688337 2481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:19.688294 2484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:19.688508 2482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:19.688723 2476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:19.689018 2476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:19.690202 2476 hybrid_clock.cc:648] HybridClock initialized: now 1790543539690187 us; error 34 us; skew 500 ppm
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:19.692800 2476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:19.693243 2476 webserver.cc:460] Webserver started at http://127.0.98.3:37119/ using document root <none> and password file <none>
I20260927 21:12:19.693490 2476 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:19.693533 2476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:19.693625 2476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:19.694650 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "528582be4be0418289001b3643411d4c"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "cdaf8a83bedfbb5ef2793049908cb374"
server_key_iv: "b26fd3ef3abdeb6e3a60000f240110a7"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.694957 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "528582be4be0418289001b3643411d4c"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "cdaf8a83bedfbb5ef2793049908cb374"
server_key_iv: "b26fd3ef3abdeb6e3a60000f240110a7"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.696205 2476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:19.696877 2491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.697032 2476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:19.697113 2476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "528582be4be0418289001b3643411d4c"
format_stamp: "Formatted at 2026-09-27 21:12:19 on dist-test-slave-r9x1"
server_key: "cdaf8a83bedfbb5ef2793049908cb374"
server_key_iv: "b26fd3ef3abdeb6e3a60000f240110a7"
server_key_version: "encryptionkey@0"
I20260927 21:12:19.697183 2476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:19.707219 2476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:19.709488 2476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:19.709628 2476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:19.710188 2476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:19.710235 2476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.710258 2476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:19.710269 2476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.716894 2476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38715
I20260927 21:12:19.716977 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38715 every 8 connection(s)
I20260927 21:12:19.717336 2476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:19.720798 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.710738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58291 (local address 127.0.98.62:36737)
0927 21:12:19.710807 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.710809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.710997 (+ 188us) server_negotiation.cc:407] Connection header received
0927 21:12:19.711517 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.711518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.711533 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.711613 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:19.712553 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.713378 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.715029 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.715334 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.715432 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.718207 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:19.718210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:19.718210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:19.718218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:19.719580 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.720016 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.720017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.720017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.720069 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.720278 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.720278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.720279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.720398 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:19.720444 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.720665 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.720730 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:12:19.720994 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.710683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36737 (local address 127.0.98.3:58291)
0927 21:12:19.710855 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.710897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.711405 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.711700 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.711705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.712082 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:19.712431 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.712439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.713493 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.713495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.714922 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.714925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.715440 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.715441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.715591 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.716308 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:19.716319 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:19.718100 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:19.719684 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.719692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.719694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.719934 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.720124 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.720125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.720125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.720226 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.720436 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:19.720439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:19.720565 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.720645 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.720774 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260927 21:12:19.721519 2607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36737
I20260927 21:12:19.721601 2607 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:19.721779 2607 heartbeater.cc:511] Master 127.0.98.62:36737 requested a full tablet report, sending...
I20260927 21:12:19.722096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 2476
I20260927 21:12:19.722170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:19.722160 2123 ts_manager.cc:194] Registered new tserver with Master: 528582be4be0418289001b3643411d4c (127.0.98.3:38715)
I20260927 21:12:19.722359 392 external_mini_cluster.cc:1428] Setting key e785a0a994f59174d8531a63baa6995e
I20260927 21:12:19.722640 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58291
I20260927 21:12:19.723115 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:12:19.729169 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.722922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36905 (local address 127.0.98.62:36737)
0927 21:12:19.722962 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.722964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.723205 (+ 241us) server_negotiation.cc:407] Connection header received
0927 21:12:19.723290 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.723291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.723302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.723338 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:19.723701 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.724236 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.725207 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.725516 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.725558 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.726630 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:19.726631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:19.726631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:19.726638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:19.728119 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.728450 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.728451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.728452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.728507 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.728658 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.728659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.728659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.728796 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:19.728865 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.729035 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.729099 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:12:19.729233 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.723026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36737 (local address 127.0.98.3:36905)
0927 21:12:19.723149 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.723160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.723228 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.723396 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.723396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.723582 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:19.723628 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.723630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.724340 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.724341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.725106 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.725109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.725635 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.725636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.725707 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.726146 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:19.726151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:19.726537 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:19.728207 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.728208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.728208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.728385 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.728561 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.728561 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.728562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.728609 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.728882 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:19.728883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:19.728935 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.729009 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.729120 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:19.746354 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.739309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.0.98.62:36737)
0927 21:12:19.739383 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.739385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.739395 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:19.739430 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.739431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.739442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.739492 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:19.740140 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.740695 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.741490 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.741689 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.741746 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.743985 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:19.743987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:19.743987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:19.743997 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:19.745329 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.745648 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.745649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.745649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.745702 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.745865 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.745866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.745867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.745979 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:19.746068 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.746173 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.746259 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:12:19.748006 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:19.749673 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:19.756309 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.752357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.0.98.2:39101)
0927 21:12:19.752646 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.752650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.752658 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:19.752693 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.752695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.752710 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.752784 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:19.752961 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.754151 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.755443 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.755911 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.755976 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.756037 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.756166 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.756200 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":92,"threads_started":1}
I20260927 21:12:19.756397 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.752094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39101 (local address 127.0.0.1:50030)
0927 21:12:19.752505 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.752515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.752550 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.752799 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.752799 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.752801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:19.752851 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.752853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.754250 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.754251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.755364 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.755366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.755990 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.755991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.756069 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.756083 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.756146 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.756200 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":92,"threads_started":1}
I20260927 21:12:19.756552 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.752359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.0.98.1:37973)
0927 21:12:19.752900 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.752903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.752912 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:19.752946 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.752948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.752962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.753033 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:19.753202 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.754316 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.755396 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.756107 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.756163 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.756210 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.756411 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.756450 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":98,"threads_started":1}
I20260927 21:12:19.756713 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.752090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.0.1:47746)
0927 21:12:19.752435 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.752445 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.752496 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.753095 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.753095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.753097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:19.753131 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.753133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.754395 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.754396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.755308 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.755311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.756238 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.756239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.756292 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.756295 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.756337 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.756372 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260927 21:12:19.756830 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.752271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.0.98.3:38715)
0927 21:12:19.752561 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.752566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.752576 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:19.752609 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.752611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.752623 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.752684 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:19.752868 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.753774 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.755164 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.755937 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.756004 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.756058 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.756591 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.756685 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":75,"threads_started":1}
I20260927 21:12:19.757009 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.752090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38715 (local address 127.0.0.1:52764)
0927 21:12:19.752271 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.752314 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.752394 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.752711 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.752714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.752716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:19.752790 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.752793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.753851 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.753852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.755073 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.755076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.756094 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.756095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.756165 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.756170 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.756212 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.756366 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20260927 21:12:19.757097 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 115811c30d2d4efcae1fb19d32bed27f (DEFAULT_TABLE table=test-table [id=7f7544add43b463cbb557cef0444f42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:19.757381 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 115811c30d2d4efcae1fb19d32bed27f (DEFAULT_TABLE table=test-table [id=7f7544add43b463cbb557cef0444f42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:19.757503 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115811c30d2d4efcae1fb19d32bed27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:19.757716 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115811c30d2d4efcae1fb19d32bed27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:19.760308 2630 tablet_bootstrap.cc:492] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: Bootstrap starting.
I20260927 21:12:19.760681 2631 tablet_bootstrap.cc:492] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: Bootstrap starting.
I20260927 21:12:19.760989 2630 tablet_bootstrap.cc:654] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:19.761040 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 115811c30d2d4efcae1fb19d32bed27f (DEFAULT_TABLE table=test-table [id=7f7544add43b463cbb557cef0444f42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:19.761200 2631 tablet_bootstrap.cc:654] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:19.761399 2630 log.cc:824] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:19.761410 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115811c30d2d4efcae1fb19d32bed27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:19.761528 2631 log.cc:824] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:19.762152 2630 tablet_bootstrap.cc:492] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: No bootstrap required, opened a new log
I20260927 21:12:19.762216 2630 ts_tablet_manager.cc:1397] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:12:19.762230 2631 tablet_bootstrap.cc:492] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: No bootstrap required, opened a new log
I20260927 21:12:19.762287 2631 ts_tablet_manager.cc:1397] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:19.763913 2630 raft_consensus.cc:348] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.764091 2630 raft_consensus.cc:374] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:19.764151 2630 raft_consensus.cc:729] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fe9bb937804f46b0f931e1a3a9d5a5, State: Initialized, Role: FOLLOWER
I20260927 21:12:19.764266 2630 consensus_queue.cc:260] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.764248 2631 raft_consensus.cc:348] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.764413 2631 raft_consensus.cc:374] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:19.764457 2631 raft_consensus.cc:729] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a81ac2daf34a24875d81e326f5b897, State: Initialized, Role: FOLLOWER
I20260927 21:12:19.764565 2630 ts_tablet_manager.cc:1428] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 21:12:19.764568 2631 consensus_queue.cc:260] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.764724 2467 heartbeater.cc:503] Master 127.0.98.62:36737 was elected leader, sending a full tablet report...
I20260927 21:12:19.764807 2631 ts_tablet_manager.cc:1428] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:12:19.764846 2635 tablet_bootstrap.cc:492] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: Bootstrap starting.
I20260927 21:12:19.764920 2327 heartbeater.cc:503] Master 127.0.98.62:36737 was elected leader, sending a full tablet report...
I20260927 21:12:19.765910 2635 tablet_bootstrap.cc:654] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:19.766215 2635 log.cc:824] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:19.767180 2635 tablet_bootstrap.cc:492] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: No bootstrap required, opened a new log
I20260927 21:12:19.767246 2635 ts_tablet_manager.cc:1397] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:19.768744 2635 raft_consensus.cc:348] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.768903 2635 raft_consensus.cc:374] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:19.768935 2635 raft_consensus.cc:729] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528582be4be0418289001b3643411d4c, State: Initialized, Role: FOLLOWER
I20260927 21:12:19.769048 2635 consensus_queue.cc:260] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.769320 2635 ts_tablet_manager.cc:1428] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:19.769376 2607 heartbeater.cc:503] Master 127.0.98.62:36737 was elected leader, sending a full tablet report...
I20260927 21:12:19.806635 2638 raft_consensus.cc:482] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:19.806767 2638 raft_consensus.cc:504] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.807132 2638 leader_election.cc:288] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101)
I20260927 21:12:19.810191 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.807396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46267 (local address 127.0.98.1:37973)
0927 21:12:19.807443 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.807445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.807456 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:12:19.807515 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.807515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.807527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.807592 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:19.807763 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.808335 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.809426 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.809837 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.809889 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.809898 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.810078 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.810119 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:12:19.810189 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.807319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.98.3:46267)
0927 21:12:19.807392 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.807404 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.807455 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.807606 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.807606 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.807608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:19.807676 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.807679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.808411 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.808412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.809349 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.809352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.809907 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.809908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.810013 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.810018 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.810068 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.810111 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260927 21:12:19.810251 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.807401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42481 (local address 127.0.98.2:39101)
0927 21:12:19.807443 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.807445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.807658 (+ 213us) server_negotiation.cc:407] Connection header received
0927 21:12:19.807703 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.807704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.807729 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.807775 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:19.807985 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.808501 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.809542 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.809939 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.810023 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.810033 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.810158 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.810194 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:12:19.810503 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.807327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39101 (local address 127.0.98.3:42481)
0927 21:12:19.807583 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.807593 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.807649 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.807798 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.807798 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.807809 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:19.807903 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.807905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.808581 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.808582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.809461 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.809464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.810022 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.810023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.810093 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.810097 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.810147 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.810185 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":64,"thread_start_us":128,"threads_started":1}
I20260927 21:12:19.810585 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115811c30d2d4efcae1fb19d32bed27f" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897" is_pre_election: true
I20260927 21:12:19.810770 2281 raft_consensus.cc:2454] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 0.
I20260927 21:12:19.810797 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115811c30d2d4efcae1fb19d32bed27f" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" is_pre_election: true
I20260927 21:12:19.810916 2421 raft_consensus.cc:2454] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 0.
I20260927 21:12:19.811033 2493 leader_election.cc:302] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a81ac2daf34a24875d81e326f5b897, 528582be4be0418289001b3643411d4c; no voters:
I20260927 21:12:19.811165 2638 raft_consensus.cc:2781] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:19.811224 2638 raft_consensus.cc:482] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:19.811240 2638 raft_consensus.cc:3037] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:19.811965 2638 raft_consensus.cc:504] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.812093 2638 leader_election.cc:288] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 election: Requested vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101)
W20260927 21:12:19.812196 2468 tablet.cc:2369] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:19.812237 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115811c30d2d4efcae1fb19d32bed27f" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897"
I20260927 21:12:19.812319 2281 raft_consensus.cc:3037] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:19.812584 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115811c30d2d4efcae1fb19d32bed27f" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5"
I20260927 21:12:19.812664 2421 raft_consensus.cc:3037] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:19.813221 2281 raft_consensus.cc:2454] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 1.
I20260927 21:12:19.813436 2493 leader_election.cc:302] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a81ac2daf34a24875d81e326f5b897, 528582be4be0418289001b3643411d4c; no voters:
I20260927 21:12:19.813516 2638 raft_consensus.cc:2781] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:19.813520 2421 raft_consensus.cc:2454] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 1.
I20260927 21:12:19.813678 2638 raft_consensus.cc:686] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 1 LEADER]: Becoming Leader. State: Replica: 528582be4be0418289001b3643411d4c, State: Running, Role: LEADER
I20260927 21:12:19.813804 2638 consensus_queue.cc:237] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.814687 2123 catalog_manager.cc:5495] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 528582be4be0418289001b3643411d4c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } health_report { overall_health: HEALTHY } } }
Sep 27 21:12:19 dist-test-slave-r9x1 krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:19.848716 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.842131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.0.98.62:36737)
0927 21:12:19.842216 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:19.842218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:19.842293 (+ 75us) server_negotiation.cc:407] Connection header received
0927 21:12:19.842359 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:19.842359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:19.842369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:19.842417 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:19.842869 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.843355 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.844053 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.844226 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.844279 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.846338 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:19.846339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:19.846339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:19.846345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:19.847609 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.848013 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.848014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.848014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.848061 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:19.848212 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:19.848213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:19.848214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:19.848326 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:19.848380 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:19.848517 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 21:12:19.848616 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":62}
I20260927 21:12:19.848788 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:19.842048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36737 (local address 127.0.0.1:39202)
0927 21:12:19.842216 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:19.842227 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:19.842302 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:19.842426 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:19.842427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:19.842739 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:19.842785 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.842787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.843427 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.843428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.843948 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:19.843950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:19.844283 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:19.844284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:19.844331 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:19.844855 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:19.844859 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:19.846253 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:19.847694 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.847703 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.847705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.847947 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.848113 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:19.848114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:19.848114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:19.848164 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:19.848379 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:19.848387 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:19.848433 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:12:19.848490 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 21:12:19.848685 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":85,"thread_start_us":91,"threads_started":1}
I20260927 21:12:19.850890 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39202:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:12:19.851142 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:19.852625 2541 tablet_service.cc:1467] Processing CreateTablet for tablet bffa8f989eee4e179f4d29014f32e3be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a061b29ef0c4e0d9868a152f3b665ed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:12:19.852632 2401 tablet_service.cc:1467] Processing CreateTablet for tablet bffa8f989eee4e179f4d29014f32e3be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a061b29ef0c4e0d9868a152f3b665ed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:12:19.852619 2261 tablet_service.cc:1467] Processing CreateTablet for tablet bffa8f989eee4e179f4d29014f32e3be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a061b29ef0c4e0d9868a152f3b665ed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:12:19.852774 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffa8f989eee4e179f4d29014f32e3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:19.852778 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffa8f989eee4e179f4d29014f32e3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:19.852778 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffa8f989eee4e179f4d29014f32e3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:19.854298 2631 tablet_bootstrap.cc:492] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: Bootstrap starting.
I20260927 21:12:19.854424 2635 tablet_bootstrap.cc:492] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: Bootstrap starting.
I20260927 21:12:19.854557 2630 tablet_bootstrap.cc:492] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: Bootstrap starting.
I20260927 21:12:19.854861 2635 tablet_bootstrap.cc:654] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:19.855016 2631 tablet_bootstrap.cc:654] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:19.855125 2630 tablet_bootstrap.cc:654] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:19.855569 2635 tablet_bootstrap.cc:492] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: No bootstrap required, opened a new log
I20260927 21:12:19.855628 2635 ts_tablet_manager.cc:1397] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:19.855751 2631 tablet_bootstrap.cc:492] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: No bootstrap required, opened a new log
I20260927 21:12:19.855794 2631 ts_tablet_manager.cc:1397] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:19.855783 2635 raft_consensus.cc:348] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.855870 2635 raft_consensus.cc:374] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:19.855896 2635 raft_consensus.cc:729] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528582be4be0418289001b3643411d4c, State: Initialized, Role: FOLLOWER
I20260927 21:12:19.855940 2635 consensus_queue.cc:260] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.855945 2631 raft_consensus.cc:348] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.856022 2631 raft_consensus.cc:374] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:19.856031 2635 ts_tablet_manager.cc:1428] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:19.856045 2631 raft_consensus.cc:729] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a81ac2daf34a24875d81e326f5b897, State: Initialized, Role: FOLLOWER
I20260927 21:12:19.856048 2630 tablet_bootstrap.cc:492] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: No bootstrap required, opened a new log
I20260927 21:12:19.856096 2630 ts_tablet_manager.cc:1397] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:12:19.856087 2631 consensus_queue.cc:260] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.856168 2631 ts_tablet_manager.cc:1428] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.856163 2638 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:19.856230 2638 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:19.856242 2630 raft_consensus.cc:348] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.856315 2630 raft_consensus.cc:374] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:19.856334 2630 raft_consensus.cc:729] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fe9bb937804f46b0f931e1a3a9d5a5, State: Initialized, Role: FOLLOWER
I20260927 21:12:19.856372 2630 consensus_queue.cc:260] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:19.856459 2630 ts_tablet_manager.cc:1428] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:19.856561 2636 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:19.856648 2636 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:19.856789 2634 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:19.856884 2634 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 21:12:19.904817 2328 tablet.cc:2369] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:12:19.967995 2608 tablet.cc:2369] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:20.054787 2638 raft_consensus.cc:482] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:20.054930 2638 raft_consensus.cc:504] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.055110 2638 leader_election.cc:288] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101)
I20260927 21:12:20.055351 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffa8f989eee4e179f4d29014f32e3be" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897" is_pre_election: true
I20260927 21:12:20.055338 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffa8f989eee4e179f4d29014f32e3be" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" is_pre_election: true
I20260927 21:12:20.055436 2281 raft_consensus.cc:2454] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 0.
I20260927 21:12:20.055451 2421 raft_consensus.cc:2454] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 0.
I20260927 21:12:20.055626 2494 leader_election.cc:302] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528582be4be0418289001b3643411d4c, 76fe9bb937804f46b0f931e1a3a9d5a5; no voters:
I20260927 21:12:20.055743 2638 raft_consensus.cc:2781] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:20.055784 2638 raft_consensus.cc:482] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:20.055805 2638 raft_consensus.cc:3037] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:20.056450 2638 raft_consensus.cc:504] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.056571 2638 leader_election.cc:288] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 election: Requested vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101)
I20260927 21:12:20.056725 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffa8f989eee4e179f4d29014f32e3be" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897"
I20260927 21:12:20.056754 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffa8f989eee4e179f4d29014f32e3be" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5"
I20260927 21:12:20.056816 2281 raft_consensus.cc:3037] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:20.056821 2421 raft_consensus.cc:3037] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:20.057724 2421 raft_consensus.cc:2454] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 1.
I20260927 21:12:20.057724 2281 raft_consensus.cc:2454] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 1.
I20260927 21:12:20.057971 2494 leader_election.cc:302] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528582be4be0418289001b3643411d4c, 76fe9bb937804f46b0f931e1a3a9d5a5; no voters:
I20260927 21:12:20.058163 2638 raft_consensus.cc:2781] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:20.058214 2638 raft_consensus.cc:686] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [term 1 LEADER]: Becoming Leader. State: Replica: 528582be4be0418289001b3643411d4c, State: Running, Role: LEADER
I20260927 21:12:20.058274 2638 consensus_queue.cc:237] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.058624 2640 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: TxnStatusTablet state changed. Reason: New leader 528582be4be0418289001b3643411d4c. Latest consensus state: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:20.058697 2640 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:12:20.058885 2652 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:12:20.059020 2652 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:12:20.059105 2123 catalog_manager.cc:5495] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 528582be4be0418289001b3643411d4c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } health_report { overall_health: HEALTHY } } }
I20260927 21:12:20.084236 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.081531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.0.98.3:38715)
0927 21:12:20.081582 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:20.081584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:20.081592 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:20.081672 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:20.081672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:20.081692 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:20.081747 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:20.081882 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.082388 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.083500 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.083694 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.083750 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.083943 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.084041 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.084096 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:20.084153 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 21:12:20.084171 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:12:20.084301 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.081448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38715 (local address 127.0.0.1:52776)
0927 21:12:20.081522 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:20.081535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:20.081588 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:20.081762 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:20.081763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:20.081765 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:20.081821 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.081824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.082453 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.082454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.083400 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.083403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.083761 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.083762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.083826 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.083849 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.084093 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.084099 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:12:20.084148 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:12:20.084202 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 21:12:20.086372 2421 raft_consensus.cc:1261] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Refusing update from remote peer 528582be4be0418289001b3643411d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:20.086364 2281 raft_consensus.cc:1261] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Refusing update from remote peer 528582be4be0418289001b3643411d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:20.086747 2640 consensus_queue.cc:1035] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:20.086938 2638 consensus_queue.cc:1035] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:20.088016 2634 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: TxnStatusTablet state changed. Reason: New leader 528582be4be0418289001b3643411d4c. Latest consensus state: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:20.088119 2634 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:20.089447 2640 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:20.089553 2640 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:12:20.089517 2634 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:20.089589 2634 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 76fe9bb937804f46b0f931e1a3a9d5a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:20.089879 2655 mvcc.cc:205] Tried to move back new op lower bound from 7334066340190871552 to 7334066340079067136. Current Snapshot: MvccSnapshot[applied={T|T < 7334066340190871552}]
I20260927 21:12:20.090126 2656 mvcc.cc:205] Tried to move back new op lower bound from 7334066340190871552 to 7334066340079067136. Current Snapshot: MvccSnapshot[applied={T|T < 7334066340190871552}]
I20260927 21:12:20.090623 2636 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: TxnStatusTablet state changed. Reason: New leader 528582be4be0418289001b3643411d4c. Latest consensus state: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:20.090725 2636 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:20.091327 2640 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:20.091419 2640 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 528582be4be0418289001b3643411d4c: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:12:20.092023 2636 tablet_replica.cc:439] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } }
I20260927 21:12:20.092213 2636 tablet_replica.cc:442] T bffa8f989eee4e179f4d29014f32e3be P 33a81ac2daf34a24875d81e326f5b897: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:20.096308 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.093564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.0.98.3:38715)
0927 21:12:20.093608 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:20.093610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:20.093618 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:20.093642 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:20.093643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:20.093653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:20.093707 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:20.093898 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.094360 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.095567 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.095826 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.095893 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.096086 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.096120 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.096170 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:20.096224 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 21:12:20.096247 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:12:20.102551 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.099545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45201 (local address 127.0.98.3:38715)
0927 21:12:20.099602 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:20.099603 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:20.099611 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:20.099632 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:20.099633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:20.099652 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:20.099712 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:20.099893 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.100668 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.101803 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.102007 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.102073 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.102297 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.102325 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.102358 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:20.102439 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 21:12:20.102461 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37}
I20260927 21:12:20.102626 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.099428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38715 (local address 127.0.98.3:45201)
0927 21:12:20.099498 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:20.099512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:20.099565 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:20.099716 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:20.099717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:20.099718 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:20.099833 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.099835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.100758 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.100759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.101713 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.101716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.102084 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.102085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.102155 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.102180 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.102383 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.102384 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:12:20.102436 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 21:12:20.102487 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":1}
I20260927 21:12:20.105005 2281 raft_consensus.cc:1261] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Refusing update from remote peer 528582be4be0418289001b3643411d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:20.105027 2421 raft_consensus.cc:1261] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Refusing update from remote peer 528582be4be0418289001b3643411d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:20.105311 2662 consensus_queue.cc:1035] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:20.105435 2638 consensus_queue.cc:1035] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:20.120047 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:20.120186 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:20.120985 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f: Sending DeleteTablet for 3 replicas of tablet 115811c30d2d4efcae1fb19d32bed27f
I20260927 21:12:20.121356 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 115811c30d2d4efcae1fb19d32bed27f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52764
I20260927 21:12:20.121428 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 115811c30d2d4efcae1fb19d32bed27f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50030
I20260927 21:12:20.121623 2670 tablet_replica.cc:330] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: stopping tablet replica
I20260927 21:12:20.121672 2671 tablet_replica.cc:330] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: stopping tablet replica
I20260927 21:12:20.121731 2670 raft_consensus.cc:2229] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:20.121764 2671 raft_consensus.cc:2229] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:20.121740 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 115811c30d2d4efcae1fb19d32bed27f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47746
I20260927 21:12:20.121816 2671 raft_consensus.cc:2258] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:20.121824 2670 raft_consensus.cc:2258] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:20.122313 2671 ts_tablet_manager.cc:1905] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:20.122298 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:20.122458 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:20.122893 2670 ts_tablet_manager.cc:1905] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:20.123768 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba22dde37ee4b6395689ca17a41d553 (DEFAULT_TABLE table=test-table [id=2f297d5ed8ab4472b14881a1d4d98021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:20.123976 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba22dde37ee4b6395689ca17a41d553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:20.124689 2671 ts_tablet_manager.cc:1918] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:20.124755 2671 log.cc:1196] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/115811c30d2d4efcae1fb19d32bed27f
I20260927 21:12:20.124996 2671 ts_tablet_manager.cc:1939] T 115811c30d2d4efcae1fb19d32bed27f P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting consensus metadata
I20260927 21:12:20.125069 2670 ts_tablet_manager.cc:1918] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:20.125130 2670 log.cc:1196] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/115811c30d2d4efcae1fb19d32bed27f
I20260927 21:12:20.125352 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba22dde37ee4b6395689ca17a41d553 (DEFAULT_TABLE table=test-table [id=2f297d5ed8ab4472b14881a1d4d98021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:20.125411 2630 tablet_bootstrap.cc:492] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: Bootstrap starting.
I20260927 21:12:20.125471 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba22dde37ee4b6395689ca17a41d553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:20.125787 2630 tablet_bootstrap.cc:654] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:20.126171 2110 catalog_manager.cc:4841] TS 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101): tablet 115811c30d2d4efcae1fb19d32bed27f (table test-table [id=7f7544add43b463cbb557cef0444f42a]) successfully deleted
I20260927 21:12:20.126302 2630 tablet_bootstrap.cc:492] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: No bootstrap required, opened a new log
I20260927 21:12:20.126351 2630 ts_tablet_manager.cc:1397] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:20.126477 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba22dde37ee4b6395689ca17a41d553 (DEFAULT_TABLE table=test-table [id=2f297d5ed8ab4472b14881a1d4d98021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:20.126485 2630 raft_consensus.cc:348] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.126564 2630 raft_consensus.cc:374] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:20.126583 2630 raft_consensus.cc:729] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fe9bb937804f46b0f931e1a3a9d5a5, State: Initialized, Role: FOLLOWER
I20260927 21:12:20.126587 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba22dde37ee4b6395689ca17a41d553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:20.126627 2630 consensus_queue.cc:260] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.126704 2630 ts_tablet_manager.cc:1428] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:20.125371 2670 ts_tablet_manager.cc:1939] T 115811c30d2d4efcae1fb19d32bed27f P 528582be4be0418289001b3643411d4c: Deleting consensus metadata
I20260927 21:12:20.126791 2635 tablet_bootstrap.cc:492] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: Bootstrap starting.
I20260927 21:12:20.127067 2635 tablet_bootstrap.cc:654] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:20.127087 2111 catalog_manager.cc:4841] TS 528582be4be0418289001b3643411d4c (127.0.98.3:38715): tablet 115811c30d2d4efcae1fb19d32bed27f (table test-table [id=7f7544add43b463cbb557cef0444f42a]) successfully deleted
I20260927 21:12:20.127239 2672 tablet_replica.cc:330] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: stopping tablet replica
I20260927 21:12:20.127365 2672 raft_consensus.cc:2229] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:20.127421 2672 raft_consensus.cc:2258] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:20.127983 2631 tablet_bootstrap.cc:492] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: Bootstrap starting.
I20260927 21:12:20.128113 2635 tablet_bootstrap.cc:492] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: No bootstrap required, opened a new log
I20260927 21:12:20.128154 2635 ts_tablet_manager.cc:1397] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 21:12:20.128311 2635 raft_consensus.cc:348] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.128396 2635 raft_consensus.cc:374] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:20.128417 2635 raft_consensus.cc:729] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528582be4be0418289001b3643411d4c, State: Initialized, Role: FOLLOWER
I20260927 21:12:20.128463 2635 consensus_queue.cc:260] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.128535 2635 ts_tablet_manager.cc:1428] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:20.128935 2631 tablet_bootstrap.cc:654] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:20.127929 2672 ts_tablet_manager.cc:1905] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:20.129696 2631 tablet_bootstrap.cc:492] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: No bootstrap required, opened a new log
I20260927 21:12:20.129748 2631 ts_tablet_manager.cc:1397] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:20.129886 2631 raft_consensus.cc:348] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.129953 2631 raft_consensus.cc:374] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:20.129973 2631 raft_consensus.cc:729] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a81ac2daf34a24875d81e326f5b897, State: Initialized, Role: FOLLOWER
I20260927 21:12:20.130015 2631 consensus_queue.cc:260] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.130096 2631 ts_tablet_manager.cc:1428] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:20.130859 2672 ts_tablet_manager.cc:1918] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:20.130915 2672 log.cc:1196] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/115811c30d2d4efcae1fb19d32bed27f
I20260927 21:12:20.131078 2672 ts_tablet_manager.cc:1939] T 115811c30d2d4efcae1fb19d32bed27f P 33a81ac2daf34a24875d81e326f5b897: Deleting consensus metadata
I20260927 21:12:20.131636 2109 catalog_manager.cc:4841] TS 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973): tablet 115811c30d2d4efcae1fb19d32bed27f (table test-table [id=7f7544add43b463cbb557cef0444f42a]) successfully deleted
I20260927 21:12:20.141549 2634 raft_consensus.cc:482] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:20.141656 2634 raft_consensus.cc:504] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.141983 2634 leader_election.cc:288] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 528582be4be0418289001b3643411d4c (127.0.98.3:38715)
I20260927 21:12:20.145319 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.142256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41807 (local address 127.0.98.3:38715)
0927 21:12:20.142316 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:20.142318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:20.142387 (+ 69us) server_negotiation.cc:407] Connection header received
0927 21:12:20.142448 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:20.142448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:20.142458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:20.142520 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:20.142859 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.143552 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.144563 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.144958 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.145026 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.145038 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:20.145184 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 21:12:20.145217 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:12:20.145351 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.142157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38715 (local address 127.0.98.2:41807)
0927 21:12:20.142306 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:20.142319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:20.142388 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:20.142686 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:20.142687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:20.142705 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:20.142776 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.142778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.143633 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.143634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.144474 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.144476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.145036 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.145037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.145112 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.145117 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:12:20.145174 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 21:12:20.145216 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":39,"thread_start_us":59,"threads_started":1}
I20260927 21:12:20.145733 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba22dde37ee4b6395689ca17a41d553" candidate_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528582be4be0418289001b3643411d4c" is_pre_election: true
I20260927 21:12:20.145859 2561 raft_consensus.cc:2454] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fe9bb937804f46b0f931e1a3a9d5a5 in term 0.
I20260927 21:12:20.146018 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.142356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43917 (local address 127.0.98.1:37973)
0927 21:12:20.142398 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:20.142399 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:20.142526 (+ 127us) server_negotiation.cc:407] Connection header received
0927 21:12:20.142560 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:20.142561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:20.142570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:20.142632 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:20.142762 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.143519 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.144922 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.145480 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.145551 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.145562 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:20.145899 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 21:12:20.145950 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:12:20.146042 2355 leader_election.cc:302] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528582be4be0418289001b3643411d4c, 76fe9bb937804f46b0f931e1a3a9d5a5; no voters:
I20260927 21:12:20.146225 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.142164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.98.2:43917)
0927 21:12:20.142448 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:20.142458 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:20.142516 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:20.142635 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:20.142636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:20.142637 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:20.142695 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.142697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.143617 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.143619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.144826 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.144829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.145718 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.145719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.145796 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.145800 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:20.146100 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 21:12:20.146145 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":94,"threads_started":1}
I20260927 21:12:20.146224 2634 raft_consensus.cc:2781] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:20.146400 2634 raft_consensus.cc:482] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:20.146426 2634 raft_consensus.cc:3037] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:20.146620 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba22dde37ee4b6395689ca17a41d553" candidate_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897" is_pre_election: true
I20260927 21:12:20.146698 2281 raft_consensus.cc:2454] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fe9bb937804f46b0f931e1a3a9d5a5 in term 0.
I20260927 21:12:20.147228 2634 raft_consensus.cc:504] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.147339 2634 leader_election.cc:288] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 528582be4be0418289001b3643411d4c (127.0.98.3:38715)
I20260927 21:12:20.147495 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba22dde37ee4b6395689ca17a41d553" candidate_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897"
I20260927 21:12:20.147562 2281 raft_consensus.cc:3037] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:20.147516 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba22dde37ee4b6395689ca17a41d553" candidate_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528582be4be0418289001b3643411d4c"
I20260927 21:12:20.147594 2561 raft_consensus.cc:3037] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:20.148150 2281 raft_consensus.cc:2454] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fe9bb937804f46b0f931e1a3a9d5a5 in term 1.
I20260927 21:12:20.148325 2353 leader_election.cc:302] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a81ac2daf34a24875d81e326f5b897, 76fe9bb937804f46b0f931e1a3a9d5a5; no voters:
I20260927 21:12:20.148411 2634 raft_consensus.cc:2781] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:20.148473 2561 raft_consensus.cc:2454] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fe9bb937804f46b0f931e1a3a9d5a5 in term 1.
I20260927 21:12:20.148551 2634 raft_consensus.cc:686] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 76fe9bb937804f46b0f931e1a3a9d5a5, State: Running, Role: LEADER
I20260927 21:12:20.148620 2634 consensus_queue.cc:237] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:20.149379 2123 catalog_manager.cc:5495] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } health_report { overall_health: UNKNOWN } } }
I20260927 21:12:20.161955 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.159214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.0.98.2:39101)
0927 21:12:20.159266 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:20.159268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:20.159275 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:20.159305 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:20.159306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:20.159323 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:20.159398 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:20.159548 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.160038 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.161110 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.161372 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.161421 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.161613 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.161724 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.161806 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:20.161860 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 21:12:20.161883 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:12:20.167459 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.164721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54135 (local address 127.0.98.3:38715)
0927 21:12:20.164766 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:20.164768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:20.164832 (+ 64us) server_negotiation.cc:407] Connection header received
0927 21:12:20.164898 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:20.164899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:20.164908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:20.164952 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:20.165083 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.165668 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.166793 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.166991 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.167057 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.167253 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.167281 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.167311 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:20.167381 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 21:12:20.167398 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:12:20.167825 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.164621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38715 (local address 127.0.98.2:54135)
0927 21:12:20.164775 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:20.164787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:20.164846 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:20.164956 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:20.164957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:20.164958 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:20.165026 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.165028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.165736 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.165738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.166693 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.166696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.167088 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.167089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.167151 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.167171 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.167322 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.167323 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:12:20.167366 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 21:12:20.167415 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260927 21:12:20.170833 2561 raft_consensus.cc:1261] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Refusing update from remote peer 76fe9bb937804f46b0f931e1a3a9d5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:20.170825 2280 raft_consensus.cc:1261] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Refusing update from remote peer 76fe9bb937804f46b0f931e1a3a9d5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:20.171084 2634 consensus_queue.cc:1035] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:20.171218 2678 consensus_queue.cc:1035] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:20.173210 2660 mvcc.cc:205] Tried to move back new op lower bound from 7334066340537765888 to 7334066340449345536. Current Snapshot: MvccSnapshot[applied={T|T < 7334066340537765888}]
I20260927 21:12:20.180006 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.177126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39101 (local address 127.0.98.3:55295)
0927 21:12:20.177170 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:20.177181 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:20.177219 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:20.177559 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:20.177560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:20.177561 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:20.177607 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.177608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.178473 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.178474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.179185 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:20.179188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.179603 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.179605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:20.179660 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.179665 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.179827 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.179829 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:12:20.179878 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:12:20.179928 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 21:12:20.180280 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:20.177371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55295 (local address 127.0.98.2:39101)
0927 21:12:20.177410 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:20.177411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:20.177418 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:20.177438 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:20.177439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:20.177450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:20.177510 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:20.177689 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.178381 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.179266 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:20.179514 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:20.179570 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:20.179733 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:20.179768 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:20.179807 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:20.180194 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 21:12:20.180218 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:12:20.180533 2656 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:12:21.709240 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:21.709381 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:21.710362 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f: Sending DeleteTablet for 3 replicas of tablet 3ba22dde37ee4b6395689ca17a41d553
I20260927 21:12:21.710659 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ba22dde37ee4b6395689ca17a41d553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50030
I20260927 21:12:21.710705 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ba22dde37ee4b6395689ca17a41d553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52764
I20260927 21:12:21.710868 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ba22dde37ee4b6395689ca17a41d553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47746
I20260927 21:12:21.710929 2709 tablet_replica.cc:330] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: stopping tablet replica
I20260927 21:12:21.710985 2709 raft_consensus.cc:2229] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:21.711081 2709 raft_consensus.cc:2258] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:21.711285 2711 tablet_replica.cc:330] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: stopping tablet replica
I20260927 21:12:21.711337 2711 raft_consensus.cc:2229] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:21.711340 2709 ts_tablet_manager.cc:1905] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:21.711372 2711 raft_consensus.cc:2258] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:21.712889 2710 tablet_replica.cc:330] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: stopping tablet replica
I20260927 21:12:21.712985 2709 ts_tablet_manager.cc:1918] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:12:21.713040 2710 raft_consensus.cc:2229] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:21.713109 2709 log.cc:1196] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/3ba22dde37ee4b6395689ca17a41d553
I20260927 21:12:21.713136 2710 raft_consensus.cc:2258] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:21.713223 2711 ts_tablet_manager.cc:1905] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:21.713433 2709 ts_tablet_manager.cc:1939] T 3ba22dde37ee4b6395689ca17a41d553 P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting consensus metadata
I20260927 21:12:21.713444 2710 ts_tablet_manager.cc:1905] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:21.713753 2110 catalog_manager.cc:4841] TS 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101): tablet 3ba22dde37ee4b6395689ca17a41d553 (table test-table [id=2f297d5ed8ab4472b14881a1d4d98021]) successfully deleted
I20260927 21:12:21.714845 2711 ts_tablet_manager.cc:1918] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:12:21.714915 2711 log.cc:1196] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/3ba22dde37ee4b6395689ca17a41d553
I20260927 21:12:21.714996 2710 ts_tablet_manager.cc:1918] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:12:21.715059 2710 log.cc:1196] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/3ba22dde37ee4b6395689ca17a41d553
I20260927 21:12:21.715150 2711 ts_tablet_manager.cc:1939] T 3ba22dde37ee4b6395689ca17a41d553 P 33a81ac2daf34a24875d81e326f5b897: Deleting consensus metadata
I20260927 21:12:21.715293 2710 ts_tablet_manager.cc:1939] T 3ba22dde37ee4b6395689ca17a41d553 P 528582be4be0418289001b3643411d4c: Deleting consensus metadata
I20260927 21:12:21.715485 2109 catalog_manager.cc:4841] TS 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973): tablet 3ba22dde37ee4b6395689ca17a41d553 (table test-table [id=2f297d5ed8ab4472b14881a1d4d98021]) successfully deleted
I20260927 21:12:21.715646 2111 catalog_manager.cc:4841] TS 528582be4be0418289001b3643411d4c (127.0.98.3:38715): tablet 3ba22dde37ee4b6395689ca17a41d553 (table test-table [id=2f297d5ed8ab4472b14881a1d4d98021]) successfully deleted
Sep 27 21:12:21 dist-test-slave-r9x1 krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:12:21 dist-test-slave-r9x1 krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:21.736940 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:21.729630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.0.98.62:36737)
0927 21:12:21.729792 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:21.729793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:21.729802 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:21.729828 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:21.729828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:21.729839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:21.729913 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:21.730503 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:21.731016 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:21.731826 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:21.732034 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:21.732091 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:21.734450 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:21.734452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:21.734452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:21.734460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:21.735886 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:21.736269 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:21.736270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:21.736270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:21.736328 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:21.736468 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:21.736469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:21.736469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:21.736588 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:21.736681 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:21.736796 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 21:12:21.736865 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260927 21:12:21.738684 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:21.738862 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:21.740396 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdafbeb51c0452cbe4fa49244d9c83f (DEFAULT_TABLE table=test-table [id=58748edc96264077a6b454d28811b1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:21.740396 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdafbeb51c0452cbe4fa49244d9c83f (DEFAULT_TABLE table=test-table [id=58748edc96264077a6b454d28811b1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:21.740535 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdafbeb51c0452cbe4fa49244d9c83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:21.740536 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdafbeb51c0452cbe4fa49244d9c83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:21.741431 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdafbeb51c0452cbe4fa49244d9c83f (DEFAULT_TABLE table=test-table [id=58748edc96264077a6b454d28811b1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:21.741564 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdafbeb51c0452cbe4fa49244d9c83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:21.742681 2721 tablet_bootstrap.cc:492] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: Bootstrap starting.
I20260927 21:12:21.743018 2721 tablet_bootstrap.cc:654] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:21.743199 2722 tablet_bootstrap.cc:492] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: Bootstrap starting.
I20260927 21:12:21.743244 2720 tablet_bootstrap.cc:492] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: Bootstrap starting.
I20260927 21:12:21.743574 2722 tablet_bootstrap.cc:654] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:21.743636 2721 tablet_bootstrap.cc:492] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: No bootstrap required, opened a new log
I20260927 21:12:21.743646 2720 tablet_bootstrap.cc:654] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:21.743690 2721 ts_tablet_manager.cc:1397] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:21.743810 2721 raft_consensus.cc:348] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.743893 2721 raft_consensus.cc:374] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:21.743913 2721 raft_consensus.cc:729] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528582be4be0418289001b3643411d4c, State: Initialized, Role: FOLLOWER
I20260927 21:12:21.743963 2721 consensus_queue.cc:260] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.744053 2721 ts_tablet_manager.cc:1428] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:21.744117 2722 tablet_bootstrap.cc:492] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: No bootstrap required, opened a new log
I20260927 21:12:21.744165 2722 ts_tablet_manager.cc:1397] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:21.744302 2722 raft_consensus.cc:348] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.744369 2722 raft_consensus.cc:374] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:21.744390 2722 raft_consensus.cc:729] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a81ac2daf34a24875d81e326f5b897, State: Initialized, Role: FOLLOWER
I20260927 21:12:21.744431 2722 consensus_queue.cc:260] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.744596 2722 ts_tablet_manager.cc:1428] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:21.744632 2720 tablet_bootstrap.cc:492] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: No bootstrap required, opened a new log
I20260927 21:12:21.744679 2720 ts_tablet_manager.cc:1397] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:21.744870 2720 raft_consensus.cc:348] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.745033 2720 raft_consensus.cc:374] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:21.745062 2720 raft_consensus.cc:729] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fe9bb937804f46b0f931e1a3a9d5a5, State: Initialized, Role: FOLLOWER
I20260927 21:12:21.745121 2720 consensus_queue.cc:260] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.745288 2720 ts_tablet_manager.cc:1428] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:21.963075 2687 raft_consensus.cc:482] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:21.963181 2687 raft_consensus.cc:504] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.963340 2687 leader_election.cc:288] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 528582be4be0418289001b3643411d4c (127.0.98.3:38715)
I20260927 21:12:21.963651 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdafbeb51c0452cbe4fa49244d9c83f" candidate_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897" is_pre_election: true
I20260927 21:12:21.963651 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdafbeb51c0452cbe4fa49244d9c83f" candidate_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528582be4be0418289001b3643411d4c" is_pre_election: true
I20260927 21:12:21.963766 2561 raft_consensus.cc:2454] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fe9bb937804f46b0f931e1a3a9d5a5 in term 0.
I20260927 21:12:21.963765 2280 raft_consensus.cc:2454] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fe9bb937804f46b0f931e1a3a9d5a5 in term 0.
I20260927 21:12:21.963944 2353 leader_election.cc:302] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a81ac2daf34a24875d81e326f5b897, 76fe9bb937804f46b0f931e1a3a9d5a5; no voters:
I20260927 21:12:21.964058 2687 raft_consensus.cc:2781] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:21.964103 2687 raft_consensus.cc:482] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:21.964118 2687 raft_consensus.cc:3037] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:21.965045 2687 raft_consensus.cc:504] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.965222 2687 leader_election.cc:288] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [CANDIDATE]: Term 1 election: Requested vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 528582be4be0418289001b3643411d4c (127.0.98.3:38715)
I20260927 21:12:21.965441 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdafbeb51c0452cbe4fa49244d9c83f" candidate_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528582be4be0418289001b3643411d4c"
I20260927 21:12:21.965466 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdafbeb51c0452cbe4fa49244d9c83f" candidate_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897"
I20260927 21:12:21.965546 2561 raft_consensus.cc:3037] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:21.965548 2280 raft_consensus.cc:3037] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:21.966310 2280 raft_consensus.cc:2454] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fe9bb937804f46b0f931e1a3a9d5a5 in term 1.
I20260927 21:12:21.966320 2561 raft_consensus.cc:2454] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fe9bb937804f46b0f931e1a3a9d5a5 in term 1.
I20260927 21:12:21.966498 2353 leader_election.cc:302] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a81ac2daf34a24875d81e326f5b897, 76fe9bb937804f46b0f931e1a3a9d5a5; no voters:
I20260927 21:12:21.966598 2687 raft_consensus.cc:2781] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:21.966648 2687 raft_consensus.cc:686] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 76fe9bb937804f46b0f931e1a3a9d5a5, State: Running, Role: LEADER
I20260927 21:12:21.966697 2687 consensus_queue.cc:237] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.967294 2124 catalog_manager.cc:5495] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } health_report { overall_health: UNKNOWN } } }
I20260927 21:12:21.975526 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:21.972993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.0.98.2:39101)
0927 21:12:21.973151 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:21.973153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:21.973160 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:21.973240 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:21.973241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:21.973251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:21.973373 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:21.973512 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:21.974023 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:21.974827 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:21.975042 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:21.975105 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:21.975295 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:21.975329 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:21.975391 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:21.975421 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 21:12:21.975443 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260927 21:12:21.977962 2561 raft_consensus.cc:1261] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Refusing update from remote peer 76fe9bb937804f46b0f931e1a3a9d5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:21.977999 2280 raft_consensus.cc:1261] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Refusing update from remote peer 76fe9bb937804f46b0f931e1a3a9d5a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:21.978191 2703 consensus_queue.cc:1035] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:21.978308 2687 consensus_queue.cc:1035] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:21.988590 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:21.988710 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:21.989466 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f: Sending DeleteTablet for 3 replicas of tablet 8fdafbeb51c0452cbe4fa49244d9c83f
I20260927 21:12:21.989756 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdafbeb51c0452cbe4fa49244d9c83f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50030
I20260927 21:12:21.989797 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdafbeb51c0452cbe4fa49244d9c83f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47746
I20260927 21:12:21.989799 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdafbeb51c0452cbe4fa49244d9c83f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52764
I20260927 21:12:21.989907 2711 tablet_replica.cc:330] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: stopping tablet replica
I20260927 21:12:21.989914 2709 tablet_replica.cc:330] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: stopping tablet replica
I20260927 21:12:21.989936 2710 tablet_replica.cc:330] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: stopping tablet replica
I20260927 21:12:21.989972 2709 raft_consensus.cc:2229] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:21.989972 2711 raft_consensus.cc:2229] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:21.989998 2710 raft_consensus.cc:2229] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:21.990020 2711 raft_consensus.cc:2258] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:21.990037 2709 raft_consensus.cc:2258] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:21.990041 2710 raft_consensus.cc:2258] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:21.990358 2709 ts_tablet_manager.cc:1905] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:21.990490 2711 ts_tablet_manager.cc:1905] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:21.991622 2710 ts_tablet_manager.cc:1905] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:21.991672 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:12:21.991860 2711 ts_tablet_manager.cc:1918] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:21.991878 2709 ts_tablet_manager.cc:1918] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
W20260927 21:12:21.991897 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:21.991906 2711 log.cc:1196] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/8fdafbeb51c0452cbe4fa49244d9c83f
I20260927 21:12:21.991933 2709 log.cc:1196] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/8fdafbeb51c0452cbe4fa49244d9c83f
I20260927 21:12:21.992128 2709 ts_tablet_manager.cc:1939] T 8fdafbeb51c0452cbe4fa49244d9c83f P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting consensus metadata
I20260927 21:12:21.992136 2711 ts_tablet_manager.cc:1939] T 8fdafbeb51c0452cbe4fa49244d9c83f P 33a81ac2daf34a24875d81e326f5b897: Deleting consensus metadata
I20260927 21:12:21.992445 2110 catalog_manager.cc:4841] TS 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101): tablet 8fdafbeb51c0452cbe4fa49244d9c83f (table test-table [id=58748edc96264077a6b454d28811b1a3]) successfully deleted
I20260927 21:12:21.992559 2109 catalog_manager.cc:4841] TS 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973): tablet 8fdafbeb51c0452cbe4fa49244d9c83f (table test-table [id=58748edc96264077a6b454d28811b1a3]) successfully deleted
I20260927 21:12:21.993079 2710 ts_tablet_manager.cc:1918] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:21.993137 2710 log.cc:1196] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/8fdafbeb51c0452cbe4fa49244d9c83f
I20260927 21:12:21.993395 2710 ts_tablet_manager.cc:1939] T 8fdafbeb51c0452cbe4fa49244d9c83f P 528582be4be0418289001b3643411d4c: Deleting consensus metadata
I20260927 21:12:21.993453 2261 tablet_service.cc:1467] Processing CreateTablet for tablet a37ea0209f194682807763797e90b4d3 (DEFAULT_TABLE table=test-table [id=a008ce0155fa4630995cec426a08a2d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:21.993451 2540 tablet_service.cc:1467] Processing CreateTablet for tablet a37ea0209f194682807763797e90b4d3 (DEFAULT_TABLE table=test-table [id=a008ce0155fa4630995cec426a08a2d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:21.993553 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37ea0209f194682807763797e90b4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:21.993585 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37ea0209f194682807763797e90b4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:21.993716 2111 catalog_manager.cc:4841] TS 528582be4be0418289001b3643411d4c (127.0.98.3:38715): tablet 8fdafbeb51c0452cbe4fa49244d9c83f (table test-table [id=58748edc96264077a6b454d28811b1a3]) successfully deleted
I20260927 21:12:21.994462 2722 tablet_bootstrap.cc:492] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897: Bootstrap starting.
I20260927 21:12:21.994472 2721 tablet_bootstrap.cc:492] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: Bootstrap starting.
I20260927 21:12:21.994659 2401 tablet_service.cc:1467] Processing CreateTablet for tablet a37ea0209f194682807763797e90b4d3 (DEFAULT_TABLE table=test-table [id=a008ce0155fa4630995cec426a08a2d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:21.994761 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37ea0209f194682807763797e90b4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:21.994829 2721 tablet_bootstrap.cc:654] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:21.994854 2722 tablet_bootstrap.cc:654] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:21.995388 2721 tablet_bootstrap.cc:492] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: No bootstrap required, opened a new log
I20260927 21:12:21.995443 2721 ts_tablet_manager.cc:1397] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:21.995564 2721 raft_consensus.cc:348] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.995631 2721 raft_consensus.cc:374] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:21.995649 2721 raft_consensus.cc:729] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528582be4be0418289001b3643411d4c, State: Initialized, Role: FOLLOWER
I20260927 21:12:21.995699 2721 consensus_queue.cc:260] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.995788 2721 ts_tablet_manager.cc:1428] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:21.996064 2720 tablet_bootstrap.cc:492] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: Bootstrap starting.
I20260927 21:12:21.996111 2722 tablet_bootstrap.cc:492] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897: No bootstrap required, opened a new log
I20260927 21:12:21.996152 2722 ts_tablet_manager.cc:1397] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:21.996299 2722 raft_consensus.cc:348] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.996366 2722 raft_consensus.cc:374] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:21.996384 2722 raft_consensus.cc:729] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a81ac2daf34a24875d81e326f5b897, State: Initialized, Role: FOLLOWER
I20260927 21:12:21.996402 2720 tablet_bootstrap.cc:654] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:21.996426 2722 consensus_queue.cc:260] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.996515 2722 ts_tablet_manager.cc:1428] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:21.996925 2720 tablet_bootstrap.cc:492] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: No bootstrap required, opened a new log
I20260927 21:12:21.996984 2720 ts_tablet_manager.cc:1397] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:21.997113 2720 raft_consensus.cc:348] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.997184 2720 raft_consensus.cc:374] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:21.997211 2720 raft_consensus.cc:729] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fe9bb937804f46b0f931e1a3a9d5a5, State: Initialized, Role: FOLLOWER
I20260927 21:12:21.997252 2720 consensus_queue.cc:260] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:21.997356 2720 ts_tablet_manager.cc:1428] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:22.260128 2696 raft_consensus.cc:482] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:22.260246 2696 raft_consensus.cc:504] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:22.260418 2696 leader_election.cc:288] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101)
I20260927 21:12:22.260681 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37ea0209f194682807763797e90b4d3" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" is_pre_election: true
I20260927 21:12:22.260679 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37ea0209f194682807763797e90b4d3" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897" is_pre_election: true
I20260927 21:12:22.260794 2421 raft_consensus.cc:2454] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 0.
I20260927 21:12:22.260802 2280 raft_consensus.cc:2454] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 0.
I20260927 21:12:22.260957 2493 leader_election.cc:302] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a81ac2daf34a24875d81e326f5b897, 528582be4be0418289001b3643411d4c; no voters:
I20260927 21:12:22.261078 2696 raft_consensus.cc:2781] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:22.261134 2696 raft_consensus.cc:482] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:22.261149 2696 raft_consensus.cc:3037] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:22.261965 2696 raft_consensus.cc:504] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:22.262091 2696 leader_election.cc:288] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 election: Requested vote from peers 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973), 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101)
I20260927 21:12:22.262260 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37ea0209f194682807763797e90b4d3" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a81ac2daf34a24875d81e326f5b897"
I20260927 21:12:22.262276 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37ea0209f194682807763797e90b4d3" candidate_uuid: "528582be4be0418289001b3643411d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5"
I20260927 21:12:22.262337 2280 raft_consensus.cc:3037] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:22.262343 2421 raft_consensus.cc:3037] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:22.262905 2421 raft_consensus.cc:2454] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 1.
I20260927 21:12:22.263005 2280 raft_consensus.cc:2454] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528582be4be0418289001b3643411d4c in term 1.
I20260927 21:12:22.263129 2494 leader_election.cc:302] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528582be4be0418289001b3643411d4c, 76fe9bb937804f46b0f931e1a3a9d5a5; no voters:
I20260927 21:12:22.263229 2696 raft_consensus.cc:2781] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:22.263289 2696 raft_consensus.cc:686] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 1 LEADER]: Becoming Leader. State: Replica: 528582be4be0418289001b3643411d4c, State: Running, Role: LEADER
I20260927 21:12:22.263351 2696 consensus_queue.cc:237] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } }
I20260927 21:12:22.263993 2124 catalog_manager.cc:5495] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 528582be4be0418289001b3643411d4c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "528582be4be0418289001b3643411d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "528582be4be0418289001b3643411d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } health_report { overall_health: HEALTHY } } }
I20260927 21:12:22.284015 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:22.281461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.0.98.3:38715)
0927 21:12:22.281666 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:22.281668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:22.281676 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:22.281695 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:22.281696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:22.281707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:22.281781 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:22.281927 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:22.282455 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:22.283293 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:22.283490 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:22.283551 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:22.283712 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:22.283745 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:22.283797 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:22.283899 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:12:22.283921 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
I20260927 21:12:22.286283 2421 raft_consensus.cc:1261] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Refusing update from remote peer 528582be4be0418289001b3643411d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:22.286353 2280 raft_consensus.cc:1261] T a37ea0209f194682807763797e90b4d3 P 33a81ac2daf34a24875d81e326f5b897 [term 1 FOLLOWER]: Refusing update from remote peer 528582be4be0418289001b3643411d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:22.286510 2739 consensus_queue.cc:1035] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "76fe9bb937804f46b0f931e1a3a9d5a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:22.286597 2690 consensus_queue.cc:1035] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a81ac2daf34a24875d81e326f5b897" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:12:22.290601 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:12:23.808740 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:23.808851 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:23.809756 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3aae6856c04e4e98b89dd9858edf794f: Sending DeleteTablet for 3 replicas of tablet a37ea0209f194682807763797e90b4d3
I20260927 21:12:23.810041 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet a37ea0209f194682807763797e90b4d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50030
I20260927 21:12:23.810065 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet a37ea0209f194682807763797e90b4d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47746
I20260927 21:12:23.810101 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet a37ea0209f194682807763797e90b4d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52764
I20260927 21:12:23.810262 2763 tablet_replica.cc:330] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: stopping tablet replica
I20260927 21:12:23.810319 2763 raft_consensus.cc:2229] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:23.810384 2763 raft_consensus.cc:2258] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:23.810403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 2196
I20260927 21:12:23.810485 2761 tablet_replica.cc:330] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: stopping tablet replica
I20260927 21:12:23.810534 2761 raft_consensus.cc:2229] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:23.810566 2761 raft_consensus.cc:2258] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:23.811492 2761 ts_tablet_manager.cc:1905] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:23.813362 2761 ts_tablet_manager.cc:1918] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:12:23.813458 2761 log.cc:1196] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/a37ea0209f194682807763797e90b4d3
I20260927 21:12:23.813623 2761 ts_tablet_manager.cc:1939] T a37ea0209f194682807763797e90b4d3 P 76fe9bb937804f46b0f931e1a3a9d5a5: Deleting consensus metadata
I20260927 21:12:23.813958 2110 catalog_manager.cc:4841] TS 76fe9bb937804f46b0f931e1a3a9d5a5 (127.0.98.2:39101): tablet a37ea0209f194682807763797e90b4d3 (table test-table [id=a008ce0155fa4630995cec426a08a2d2]) successfully deleted
I20260927 21:12:23.814052 2763 ts_tablet_manager.cc:1905] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:23.815618 2763 ts_tablet_manager.cc:1918] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:12:23.815686 2763 log.cc:1196] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/a37ea0209f194682807763797e90b4d3
I20260927 21:12:23.815883 2763 ts_tablet_manager.cc:1939] T a37ea0209f194682807763797e90b4d3 P 528582be4be0418289001b3643411d4c: Deleting consensus metadata
I20260927 21:12:23.816269 2111 catalog_manager.cc:4841] TS 528582be4be0418289001b3643411d4c (127.0.98.3:38715): tablet a37ea0209f194682807763797e90b4d3 (table test-table [id=a008ce0155fa4630995cec426a08a2d2]) successfully deleted
W20260927 21:12:23.817865 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37973 (error 108)
I20260927 21:12:23.817973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 2336
I20260927 21:12:23.819598 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:23.819348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37973 (local address 127.0.0.1:34020)
0927 21:12:23.819496 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:23.819552 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
W20260927 21:12:23.819726 2109 catalog_manager.cc:4584] TS 33a81ac2daf34a24875d81e326f5b897 (127.0.98.1:37973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a37ea0209f194682807763797e90b4d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37973: connect: Connection refused (error 111)
I20260927 21:12:23.823259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 2476
I20260927 21:12:23.829641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 2092
2026-09-27T21:12:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5836 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:23 dist-test-slave-r9x1 krb5kdc[2768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:23 dist-test-slave-r9x1 krb5kdc[2768](info): set up 2 sockets
Sep 27 21:12:23 dist-test-slave-r9x1 krb5kdc[2768](info): commencing operation
krb5kdc: starting...
W20260927 21:12:24.804888 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:24 dist-test-slave-r9x1 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:24.925215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33005
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:25.005206 2781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:25.005481 2781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:25.005505 2781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:25.006781 2781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:25.006834 2781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:25.006846 2781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:25.006858 2781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:25.006870 2781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:25.008538 2781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:25.008773 2781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:25.008958 2781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:25.014456 2786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:25.014421 2789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:25.014483 2787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:25.014433 2781 server_base.cc:1034] running on GCE node
I20260927 21:12:25.014914 2781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:25.015219 2781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:25.016366 2781 hybrid_clock.cc:648] HybridClock initialized: now 1790543545016345 us; error 42 us; skew 500 ppm
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:25.018903 2781 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:25.019383 2781 webserver.cc:460] Webserver started at http://127.0.98.62:35033/ using document root <none> and password file <none>
I20260927 21:12:25.019589 2781 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:25.019632 2781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:25.019765 2781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:25.020816 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "627a2f83978c450db63ba46bc88619a7"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "4bfd565911f71bd6aad6453877f64f46"
server_key_iv: "517d5d0f0bb390ae4569f5da290acd6b"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.021111 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "627a2f83978c450db63ba46bc88619a7"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "4bfd565911f71bd6aad6453877f64f46"
server_key_iv: "517d5d0f0bb390ae4569f5da290acd6b"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.022511 2781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:25.023309 2796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.023510 2781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:25.023603 2781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "627a2f83978c450db63ba46bc88619a7"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "4bfd565911f71bd6aad6453877f64f46"
server_key_iv: "517d5d0f0bb390ae4569f5da290acd6b"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.023689 2781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:25.033408 2781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:25.035439 2781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:25.035567 2781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:25.039544 2781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33005
I20260927 21:12:25.039585 2848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33005 every 8 connection(s)
I20260927 21:12:25.040112 2781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:25.040817 2849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:25.043205 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7: Bootstrap starting.
I20260927 21:12:25.043797 2849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:25.044080 2849 log.cc:824] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:25.044839 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7: No bootstrap required, opened a new log
I20260927 21:12:25.046504 2849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a2f83978c450db63ba46bc88619a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } }
I20260927 21:12:25.046659 2849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:25.046684 2849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627a2f83978c450db63ba46bc88619a7, State: Initialized, Role: FOLLOWER
I20260927 21:12:25.046818 2849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a2f83978c450db63ba46bc88619a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } }
I20260927 21:12:25.046890 2849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:25.046913 2849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:25.046962 2849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:25.047891 2849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a2f83978c450db63ba46bc88619a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } }
I20260927 21:12:25.047984 2849 leader_election.cc:302] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 627a2f83978c450db63ba46bc88619a7; no voters:
I20260927 21:12:25.048184 2849 leader_election.cc:288] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:25.048234 2852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:25.048403 2852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [term 1 LEADER]: Becoming Leader. State: Replica: 627a2f83978c450db63ba46bc88619a7, State: Running, Role: LEADER
I20260927 21:12:25.048444 2849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:25.048501 2852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a2f83978c450db63ba46bc88619a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } }
I20260927 21:12:25.048894 2854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 627a2f83978c450db63ba46bc88619a7. Latest consensus state: current_term: 1 leader_uuid: "627a2f83978c450db63ba46bc88619a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a2f83978c450db63ba46bc88619a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } } }
I20260927 21:12:25.048981 2854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:25.049194 2861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:25.049207 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "627a2f83978c450db63ba46bc88619a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627a2f83978c450db63ba46bc88619a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } } }
I20260927 21:12:25.049496 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:25.049839 2861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:25.050212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 2781
I20260927 21:12:25.050297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:25.050454 392 external_mini_cluster.cc:1428] Setting key 61d77c733bdd31fc80fc6f125ddc656c
I20260927 21:12:25.052213 2861 catalog_manager.cc:1347] Generated new cluster ID: 0c671338aeca411a8ada0e750e7ed374
I20260927 21:12:25.052273 2861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:25.059983 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:25.050943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.0.98.62:33005)
0927 21:12:25.051220 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:25.051230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:25.051255 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:12:25.051353 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:25.051359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:25.051410 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:25.051529 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:25.052405 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.053404 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.054151 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.054607 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.054709 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:25.056897 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:25.056903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:25.056903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:25.056917 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:25.058499 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:25.058825 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:25.058828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:25.058829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:25.058931 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:25.059107 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:25.059108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:25.059108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:25.059371 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:25.059526 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:25.059631 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:12:25.059774 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20260927 21:12:25.066721 2861 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:25.067452 2861 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:25.074434 2861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 627a2f83978c450db63ba46bc88619a7: Generated new TSK 0
I20260927 21:12:25.074666 2861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:25.104741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33005
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:25.190559 2875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:25.190783 2875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:25.190809 2875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:25.192059 2875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:25.192121 2875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:25.192148 2875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:25.193792 2875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:25.194039 2875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:25.194272 2875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:25.199219 2880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:25.199219 2883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:25.199335 2875 server_base.cc:1034] running on GCE node
W20260927 21:12:25.199235 2881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:25.199703 2875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:25.199944 2875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:25.201087 2875 hybrid_clock.cc:648] HybridClock initialized: now 1790543545201069 us; error 36 us; skew 500 ppm
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:25.203601 2875 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:25.204044 2875 webserver.cc:460] Webserver started at http://127.0.98.1:38055/ using document root <none> and password file <none>
I20260927 21:12:25.204269 2875 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:25.204327 2875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:25.204455 2875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:25.205561 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "7bc2dac0ac9c42f48b2130f11c9bbbda"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "9e1ed6ee03786cc7979ca59ead9dbfb3"
server_key_iv: "b7f92113a2127432301562bec971f103"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.205904 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7bc2dac0ac9c42f48b2130f11c9bbbda"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "9e1ed6ee03786cc7979ca59ead9dbfb3"
server_key_iv: "b7f92113a2127432301562bec971f103"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.207252 2875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:25.208045 2890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.208241 2875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:25.208348 2875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "7bc2dac0ac9c42f48b2130f11c9bbbda"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "9e1ed6ee03786cc7979ca59ead9dbfb3"
server_key_iv: "b7f92113a2127432301562bec971f103"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.208413 2875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:25.215216 2875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:25.217715 2875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:25.217867 2875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:25.218091 2875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:25.218400 2875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:25.218436 2875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.218465 2875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:25.218489 2875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.223871 2875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44735
I20260927 21:12:25.223941 3003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44735 every 8 connection(s)
I20260927 21:12:25.224261 2875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:12:25.229580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 2875
I20260927 21:12:25.229663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:25.229822 392 external_mini_cluster.cc:1428] Setting key b434fcc4295246edbdb68fb487b79599
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:25.234793 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:25.225032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41231 (local address 127.0.98.62:33005)
0927 21:12:25.225086 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:25.225089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:25.225320 (+ 231us) server_negotiation.cc:407] Connection header received
0927 21:12:25.226332 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:25.226333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:25.226351 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:25.226417 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:25.227122 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.227689 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.228716 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.228928 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.228985 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:25.231583 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:25.231585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:25.231585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:25.231594 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:25.233108 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:25.233629 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:25.233631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:25.233631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:25.233702 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:25.233954 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:25.233955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:25.233955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:25.234118 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:25.234191 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:25.234589 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 21:12:25.234693 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:12:25.234828 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:25.225009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33005 (local address 127.0.98.1:41231)
0927 21:12:25.225156 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:25.225194 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:25.226179 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:25.226543 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:25.226549 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:25.226844 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:25.227040 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:25.227044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.227774 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.227775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:25.228633 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:25.228636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.228995 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.228995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:25.229093 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:25.229659 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:25.229670 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:25.231477 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:25.233227 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:25.233235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:25.233237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:25.233534 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:25.233774 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:25.233775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:25.233775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:25.233892 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:25.234221 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:25.234230 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:25.234400 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 21:12:25.234498 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:12:25.234646 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260927 21:12:25.235312 3004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33005
I20260927 21:12:25.235419 3004 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:25.235603 3004 heartbeater.cc:511] Master 127.0.98.62:33005 requested a full tablet report, sending...
I20260927 21:12:25.236135 2813 ts_manager.cc:194] Registered new tserver with Master: 7bc2dac0ac9c42f48b2130f11c9bbbda (127.0.98.1:44735)
I20260927 21:12:25.236747 2813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41231
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:25.257045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33005
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:25.339618 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:25.339852 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:25.339887 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:25.341151 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:25.341239 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:25.341303 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:25.342975 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:25.343256 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:25.343489 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:25.348934 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:25.348937 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:25.349035 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:25.348974 3009 server_base.cc:1034] running on GCE node
I20260927 21:12:25.349647 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:25.349900 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:25.351064 3009 hybrid_clock.cc:648] HybridClock initialized: now 1790543545351050 us; error 35 us; skew 500 ppm
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:25.353931 3009 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:25.354460 3009 webserver.cc:460] Webserver started at http://127.0.98.2:40893/ using document root <none> and password file <none>
I20260927 21:12:25.354676 3009 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:25.354729 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:25.354861 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:25.355940 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "030ffb30fd044280814487362780bd7f"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "10babab5a03db29d25b56de92220e863"
server_key_iv: "8feba9728ad62fba4be549ec3d921873"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.356271 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "030ffb30fd044280814487362780bd7f"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "10babab5a03db29d25b56de92220e863"
server_key_iv: "8feba9728ad62fba4be549ec3d921873"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.357620 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:25.358309 3024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.358682 3009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:25.358786 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "030ffb30fd044280814487362780bd7f"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "10babab5a03db29d25b56de92220e863"
server_key_iv: "8feba9728ad62fba4be549ec3d921873"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.358861 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:25.371877 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:25.374308 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:25.374440 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:25.374630 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:25.375020 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:25.375069 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.375093 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:25.375104 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.381032 3009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37309
I20260927 21:12:25.381105 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37309 every 8 connection(s)
I20260927 21:12:25.381495 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:12:25.382364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 3009
I20260927 21:12:25.382453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:25.382644 392 external_mini_cluster.cc:1428] Setting key 3a90909f8a1798b70f9f47c3080ac249
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:12:25.393065 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:25.382878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44783 (local address 127.0.98.62:33005)
0927 21:12:25.383288 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:25.383291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:25.383572 (+ 281us) server_negotiation.cc:407] Connection header received
0927 21:12:25.384561 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:25.384563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:25.384577 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:25.384653 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:25.385378 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.385955 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.387127 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.387407 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.387486 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:25.389970 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:25.389972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:25.389972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:25.389981 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:25.391557 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:25.392031 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:25.392032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:25.392032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:25.392092 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:25.392391 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:25.392392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:25.392393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:25.392529 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:25.392592 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:25.392911 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 21:12:25.392993 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260927 21:12:25.393175 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:25.382931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33005 (local address 127.0.98.2:44783)
0927 21:12:25.383377 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:25.383431 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:25.384436 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:25.384739 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:25.384743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:25.385026 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:25.385244 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:25.385249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.386056 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.386057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:25.387007 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:25.387010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.387499 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.387500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:25.387623 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:25.388162 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:25.388172 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:25.389846 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:25.391661 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:25.391669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:25.391672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:25.391933 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:25.392154 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:25.392154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:25.392154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:25.392294 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:25.392599 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:25.392605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:25.392752 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 21:12:25.392846 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:12:25.392985 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":102,"threads_started":1}
I20260927 21:12:25.393692 3138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33005
I20260927 21:12:25.393812 3138 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 21:12:25.394023 3138 heartbeater.cc:511] Master 127.0.98.62:33005 requested a full tablet report, sending...
I20260927 21:12:25.394434 2813 ts_manager.cc:194] Registered new tserver with Master: 030ffb30fd044280814487362780bd7f (127.0.98.2:37309)
I20260927 21:12:25.394918 2813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:25.412865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33005
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:25.496299 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:25.496528 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:25.496562 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:25.497865 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:25.497918 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:25.497957 3143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:25.499606 3143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:25.499889 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:25.500123 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:12:25.505506 3143 server_base.cc:1034] running on GCE node
W20260927 21:12:25.505543 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:25.505553 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:25.505553 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:25.505957 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:25.506237 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:25.507413 3143 hybrid_clock.cc:648] HybridClock initialized: now 1790543545507382 us; error 48 us; skew 500 ppm
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:25.509974 3143 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:25.510460 3143 webserver.cc:460] Webserver started at http://127.0.98.3:40603/ using document root <none> and password file <none>
I20260927 21:12:25.510690 3143 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:25.510752 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:25.510896 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:25.512008 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "e47381a5d1c64816b25f31c072e64452"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "225bc73cb7a52aba8c31918a0a335016"
server_key_iv: "6b66489a641e409d5bc81fe2cc8bad90"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.512344 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e47381a5d1c64816b25f31c072e64452"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "225bc73cb7a52aba8c31918a0a335016"
server_key_iv: "6b66489a641e409d5bc81fe2cc8bad90"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.513729 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:25.514808 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.515204 3143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:25.515363 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "e47381a5d1c64816b25f31c072e64452"
format_stamp: "Formatted at 2026-09-27 21:12:25 on dist-test-slave-r9x1"
server_key: "225bc73cb7a52aba8c31918a0a335016"
server_key_iv: "6b66489a641e409d5bc81fe2cc8bad90"
server_key_version: "encryptionkey@0"
I20260927 21:12:25.515590 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:25.534101 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:25.536983 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:25.537140 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:25.537428 3143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:25.537746 3143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:25.537781 3143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.537811 3143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:25.537834 3143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:25.543516 3143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34443
I20260927 21:12:25.543609 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34443 every 8 connection(s)
I20260927 21:12:25.543901 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:25.548264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 3143
I20260927 21:12:25.548370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:25.548558 392 external_mini_cluster.cc:1428] Setting key 0871ed169d8f0090a61bbba020197a3c
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:25.554167 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:25.545138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58755 (local address 127.0.98.62:33005)
0927 21:12:25.545213 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:25.545215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:25.545230 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:12:25.546211 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:25.546212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:25.546227 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:25.546276 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:25.546941 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.547444 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.548766 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.549029 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.549091 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:25.551350 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:25.551352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:25.551353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:25.551361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:25.552643 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:25.553051 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:25.553053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:25.553053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:25.553106 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:25.553408 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:25.553409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:25.553409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:25.553531 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:25.553589 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:25.553880 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 21:12:25.553965 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:12:25.554632 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:25.544784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33005 (local address 127.0.98.3:58755)
0927 21:12:25.544937 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:25.544993 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:25.546089 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:25.546354 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:25.546359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:25.546643 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:25.546847 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:25.546852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.547536 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.547537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:25.548681 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:25.548684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:25.549097 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:25.549099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:25.549240 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:25.549832 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:25.549841 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:25.551242 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:25.552732 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:25.552740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:25.552743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:25.552979 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:25.553190 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:25.553191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:25.553191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:25.553348 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:25.553590 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:25.553596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:25.553751 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:12:25.553838 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:12:25.554012 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260927 21:12:25.555574 3272 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33005
I20260927 21:12:25.555698 3272 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:25.555900 3272 heartbeater.cc:511] Master 127.0.98.62:33005 requested a full tablet report, sending...
I20260927 21:12:25.556325 2813 ts_manager.cc:194] Registered new tserver with Master: e47381a5d1c64816b25f31c072e64452 (127.0.98.3:34443)
I20260927 21:12:25.556718 2813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58755
I20260927 21:12:25.560501 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:12:25.571563 2871 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:25.570549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.0.98.62:33005)
0927 21:12:25.570601 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:25.570603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:25.570672 (+ 69us) server_negotiation.cc:407] Connection header received
0927 21:12:25.570723 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:25.570724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:25.570749 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:25.570832 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:25.571485 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34446: BlockingRecv error: recv got EOF from 127.0.0.1:34446 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:12:25.572443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 2875
I20260927 21:12:25.576387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 3009
I20260927 21:12:25.580468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 3143
I20260927 21:12:25.584143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 2781
2026-09-27T21:12:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1753 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[3284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[3284](info): set up 2 sockets
Sep 27 21:12:25 dist-test-slave-r9x1 krb5kdc[3284](info): commencing operation
krb5kdc: starting...
W20260927 21:12:26.557331 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:26 dist-test-slave-r9x1 krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:26.687871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38929
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:26.776134 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:26.776400 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:26.776434 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:26.777822 3297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:26.777879 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:26.777897 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:26.777913 3297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:26.777928 3297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:26.779649 3297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:26.779948 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:26.780180 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:26.785944 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:26.785930 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:26.785930 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:26.786437 3297 server_base.cc:1034] running on GCE node
I20260927 21:12:26.786742 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:26.787092 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:26.788323 3297 hybrid_clock.cc:648] HybridClock initialized: now 1790543546788295 us; error 54 us; skew 500 ppm
Sep 27 21:12:26 dist-test-slave-r9x1 krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:26.795146 3297 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:26.795895 3297 webserver.cc:460] Webserver started at http://127.0.98.62:43725/ using document root <none> and password file <none>
I20260927 21:12:26.796181 3297 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:26.796240 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:26.796386 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:26.797888 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "58f32445a3704fc58c9715d571386ccf"
format_stamp: "Formatted at 2026-09-27 21:12:26 on dist-test-slave-r9x1"
server_key: "4be4627ea65c4b1430a22379a3725c2d"
server_key_iv: "3cad0ade70b53f273f1e767d17c7f1d6"
server_key_version: "encryptionkey@0"
I20260927 21:12:26.798293 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "58f32445a3704fc58c9715d571386ccf"
format_stamp: "Formatted at 2026-09-27 21:12:26 on dist-test-slave-r9x1"
server_key: "4be4627ea65c4b1430a22379a3725c2d"
server_key_iv: "3cad0ade70b53f273f1e767d17c7f1d6"
server_key_version: "encryptionkey@0"
I20260927 21:12:26.799762 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:26.800681 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:26.800885 3297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:26.800989 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "58f32445a3704fc58c9715d571386ccf"
format_stamp: "Formatted at 2026-09-27 21:12:26 on dist-test-slave-r9x1"
server_key: "4be4627ea65c4b1430a22379a3725c2d"
server_key_iv: "3cad0ade70b53f273f1e767d17c7f1d6"
server_key_version: "encryptionkey@0"
I20260927 21:12:26.801064 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:26.812351 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:26.815064 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:26.815275 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:26.819815 3297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38929
I20260927 21:12:26.819898 3364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38929 every 8 connection(s)
I20260927 21:12:26.820236 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:26.821336 3365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:26.823796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 3297
I20260927 21:12:26.823901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:26.824081 392 external_mini_cluster.cc:1428] Setting key 61ce48548c76613e1a88095389587607
I20260927 21:12:26.825131 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf: Bootstrap starting.
I20260927 21:12:26.826295 3365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:26.826715 3365 log.cc:824] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:26.827509 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf: No bootstrap required, opened a new log
I20260927 21:12:26.830042 3365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f32445a3704fc58c9715d571386ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } }
I20260927 21:12:26.830247 3365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:26.830281 3365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f32445a3704fc58c9715d571386ccf, State: Initialized, Role: FOLLOWER
I20260927 21:12:26.830415 3365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f32445a3704fc58c9715d571386ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } }
I20260927 21:12:26.830544 3365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:26.830614 3365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:26.830665 3365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:26.831390 3365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f32445a3704fc58c9715d571386ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } }
I20260927 21:12:26.831503 3365 leader_election.cc:302] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58f32445a3704fc58c9715d571386ccf; no voters:
I20260927 21:12:26.831673 3365 leader_election.cc:288] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:26.831738 3370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:26.831940 3365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:26.831962 3370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [term 1 LEADER]: Becoming Leader. State: Replica: 58f32445a3704fc58c9715d571386ccf, State: Running, Role: LEADER
I20260927 21:12:26.832059 3370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f32445a3704fc58c9715d571386ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } }
I20260927 21:12:26.832408 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58f32445a3704fc58c9715d571386ccf. Latest consensus state: current_term: 1 leader_uuid: "58f32445a3704fc58c9715d571386ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f32445a3704fc58c9715d571386ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } } }
I20260927 21:12:26.832485 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [sys.catalog]: This master's current role is: LEADER
Sep 27 21:12:26 dist-test-slave-r9x1 krb5kdc[3284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:26.832595 3371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58f32445a3704fc58c9715d571386ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58f32445a3704fc58c9715d571386ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38929 } } }
I20260927 21:12:26.832729 3371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf [sys.catalog]: This master's current role is: LEADER
W20260927 21:12:26.834136 3385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:12:26.834193 3385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:12:26.834270 3376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:26.834444 3376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:26.836226 3376 catalog_manager.cc:1347] Generated new cluster ID: d11d9a7194e44f99857492974dc0efd5
I20260927 21:12:26.836283 3376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:26.837832 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:26.824983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.0.98.62:38929)
0927 21:12:26.825292 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:26.825307 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:26.825353 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:12:26.826330 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:26.826346 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:26.826452 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:26.826670 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:26.827528 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:26.828912 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:26.829774 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:26.830400 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:26.830507 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:26.833165 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:26.833171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:26.833171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:26.833185 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:26.835911 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:26.836396 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:26.836400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:26.836400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:26.836509 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:26.836717 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:26.836718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:26.836718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:26.837062 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:26.837239 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:26.837446 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 21:12:26.837607 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":132,"threads_started":1}
I20260927 21:12:26.839489 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:12:26.846824 3376 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:26.847391 3376 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:26.851245 3376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58f32445a3704fc58c9715d571386ccf: Generated new TSK 0
I20260927 21:12:26.851441 3376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:12:26.857182 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:26.852832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.0.98.62:38929)
0927 21:12:26.852871 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:26.852872 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:26.853118 (+ 246us) server_negotiation.cc:407] Connection header received
0927 21:12:26.853189 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:26.853189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:26.853199 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:26.853250 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:26.853940 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:26.854451 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:26.855590 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:26.855808 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:26.855868 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:26.856731 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:26.856733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:26.856733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:26.856934 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:26.856952 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:26.856958 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:26.857010 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:26.857088 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 21:12:26.857119 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:12:26.860085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 3297
2026-09-27T21:12:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1275 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:26 dist-test-slave-r9x1 krb5kdc[3398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:26 dist-test-slave-r9x1 krb5kdc[3398](info): set up 2 sockets
Sep 27 21:12:26 dist-test-slave-r9x1 krb5kdc[3398](info): commencing operation
krb5kdc: starting...
W20260927 21:12:27.828902 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:27 dist-test-slave-r9x1 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:27.952066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46631
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:28.034677 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:28.034914 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:28.034937 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:28.036158 3411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:28.036199 3411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:28.036211 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:28.036223 3411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:28.036233 3411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:28.037765 3411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:28.037978 3411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:28.038158 3411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:28.042745 3419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:28.042745 3416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:28.042879 3411 server_base.cc:1034] running on GCE node
W20260927 21:12:28.042953 3417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:28.043170 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:28.043475 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:28.044729 3411 hybrid_clock.cc:648] HybridClock initialized: now 1790543548044699 us; error 30 us; skew 500 ppm
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:28.047693 3411 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:28.048154 3411 webserver.cc:460] Webserver started at http://127.0.98.62:42829/ using document root <none> and password file <none>
I20260927 21:12:28.048396 3411 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:28.048447 3411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:28.048573 3411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:28.049716 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "371409e3310a46039413cfcbd5ceff0f"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "543eff6ef9c1f3264044e5adec889695"
server_key_iv: "27103226e92acab5be5a84e185aff7b8"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.050047 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "371409e3310a46039413cfcbd5ceff0f"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "543eff6ef9c1f3264044e5adec889695"
server_key_iv: "27103226e92acab5be5a84e185aff7b8"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.051393 3411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:28.052201 3426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.052404 3411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:28.052493 3411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "371409e3310a46039413cfcbd5ceff0f"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "543eff6ef9c1f3264044e5adec889695"
server_key_iv: "27103226e92acab5be5a84e185aff7b8"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.052580 3411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:28.067932 3411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:28.069929 3411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:28.070075 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:28.073848 3411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46631
I20260927 21:12:28.073885 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46631 every 8 connection(s)
I20260927 21:12:28.074229 3411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:28.075152 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:28.077211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 3411
I20260927 21:12:28.077333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:28.077509 392 external_mini_cluster.cc:1428] Setting key 7e14d544d3ebd90c6a6ecf87c6a2bcbf
I20260927 21:12:28.078859 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f: Bootstrap starting.
I20260927 21:12:28.079604 3479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:28.079897 3479 log.cc:824] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:28.080741 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f: No bootstrap required, opened a new log
I20260927 21:12:28.083303 3479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371409e3310a46039413cfcbd5ceff0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46631 } }
I20260927 21:12:28.083519 3479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:28.083572 3479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371409e3310a46039413cfcbd5ceff0f, State: Initialized, Role: FOLLOWER
I20260927 21:12:28.083711 3479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371409e3310a46039413cfcbd5ceff0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46631 } }
I20260927 21:12:28.083793 3479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:28.083817 3479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:28.083843 3479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:28.084419 3479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371409e3310a46039413cfcbd5ceff0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46631 } }
I20260927 21:12:28.084507 3479 leader_election.cc:302] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 371409e3310a46039413cfcbd5ceff0f; no voters:
I20260927 21:12:28.084671 3479 leader_election.cc:288] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:28.084874 3484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:28.084970 3479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:28.085139 3484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [term 1 LEADER]: Becoming Leader. State: Replica: 371409e3310a46039413cfcbd5ceff0f, State: Running, Role: LEADER
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:28.085309 3484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371409e3310a46039413cfcbd5ceff0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46631 } }
I20260927 21:12:28.086150 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "371409e3310a46039413cfcbd5ceff0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371409e3310a46039413cfcbd5ceff0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46631 } } }
I20260927 21:12:28.086256 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:28.086517 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 371409e3310a46039413cfcbd5ceff0f. Latest consensus state: current_term: 1 leader_uuid: "371409e3310a46039413cfcbd5ceff0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371409e3310a46039413cfcbd5ceff0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46631 } } }
I20260927 21:12:28.086530 3496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:28.087280 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:28.087945 3496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:28.090180 3496 catalog_manager.cc:1347] Generated new cluster ID: 284d4e492d6b4c2baa192e2708dce616
I20260927 21:12:28.090276 3496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:28.090338 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:28.078099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.0.98.62:46631)
0927 21:12:28.078311 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:28.078322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:28.078358 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:12:28.078898 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:28.078908 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:28.078993 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:28.079210 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:28.080169 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.081593 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.082677 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.083116 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.083232 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:28.085895 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:28.085900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:28.085901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:28.085917 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:28.088310 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.088690 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.088694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.088694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.088783 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.088988 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.088989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.088989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.089335 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:28.089492 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:28.089931 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 21:12:28.090065 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":112,"threads_started":1}
I20260927 21:12:28.095248 3496 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:28.096012 3496 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:28.104347 3496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 371409e3310a46039413cfcbd5ceff0f: Generated new TSK 0
I20260927 21:12:28.104606 3496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:28.137199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46631
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:28.248944 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:28.249171 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:28.249197 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:28.250577 3505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:28.250643 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:28.250674 3505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:28.252388 3505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:28.252679 3505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:28.252925 3505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:28.258172 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:28.258145 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:28.258145 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:28.258354 3505 server_base.cc:1034] running on GCE node
I20260927 21:12:28.258665 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:28.258977 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:28.260134 3505 hybrid_clock.cc:648] HybridClock initialized: now 1790543548260112 us; error 36 us; skew 500 ppm
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:28.262953 3505 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:28.263422 3505 webserver.cc:460] Webserver started at http://127.0.98.1:34623/ using document root <none> and password file <none>
I20260927 21:12:28.263649 3505 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:28.263700 3505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:28.263826 3505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:28.264899 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "a293004ddf9f4344b923b69c6dcc58c1"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "febf201334d45d49a2b073b88ff2865b"
server_key_iv: "1be9eedc4bdfe378944fb72000367d00"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.265249 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a293004ddf9f4344b923b69c6dcc58c1"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "febf201334d45d49a2b073b88ff2865b"
server_key_iv: "1be9eedc4bdfe378944fb72000367d00"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.266616 3505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:28.267386 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.267544 3505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:28.267639 3505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "a293004ddf9f4344b923b69c6dcc58c1"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "febf201334d45d49a2b073b88ff2865b"
server_key_iv: "1be9eedc4bdfe378944fb72000367d00"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.267706 3505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:28.278668 3505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:28.280537 3505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:28.280680 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:28.280910 3505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:28.281209 3505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:28.281242 3505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.281299 3505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:28.281324 3505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.287024 3505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40607
I20260927 21:12:28.287264 3633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40607 every 8 connection(s)
I20260927 21:12:28.287415 3505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:12:28.293150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 3505
I20260927 21:12:28.293244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:28.293437 392 external_mini_cluster.cc:1428] Setting key d4950a391efe7763889a5992a5d8ac71
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:28.297617 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:28.288216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59893 (local address 127.0.98.62:46631)
0927 21:12:28.288280 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:28.288282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:28.288506 (+ 224us) server_negotiation.cc:407] Connection header received
0927 21:12:28.289014 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:28.289015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:28.289026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:28.289076 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:28.290116 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.290627 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.291616 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.291832 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.291895 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:28.294609 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:28.294611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:28.294611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:28.294618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:28.296008 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.296562 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.296563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.296563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.296623 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.296920 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.296920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.296921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.297043 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:28.297100 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:28.297418 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 21:12:28.297519 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:12:28.297819 3636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:28.288197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46631 (local address 127.0.98.1:59893)
0927 21:12:28.288361 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:28.288424 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:28.288920 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:28.289152 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:28.289156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:28.289796 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:28.290020 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:28.290025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.290719 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.290720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:28.291535 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:28.291537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.291905 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.291906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:28.292036 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:28.292567 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:28.292576 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:28.294455 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:28.296098 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:28.296107 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:28.296110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:28.296463 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:28.296721 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:28.296722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:28.296722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:28.296854 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:28.297104 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:28.297109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:28.297255 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 21:12:28.297394 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 21:12:28.297577 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260927 21:12:28.298343 3634 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46631
I20260927 21:12:28.298462 3634 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:28.298684 3634 heartbeater.cc:511] Master 127.0.98.62:46631 requested a full tablet report, sending...
I20260927 21:12:28.299291 3443 ts_manager.cc:194] Registered new tserver with Master: a293004ddf9f4344b923b69c6dcc58c1 (127.0.98.1:40607)
I20260927 21:12:28.299932 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59893
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:28.321657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46631
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:28.407907 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:28.408147 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:28.408174 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:28.409544 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:28.409600 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:28.409634 3639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:28.411239 3639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:28.411502 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:28.411883 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:28.417132 3647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:28.417111 3645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:28.417423 3639 server_base.cc:1034] running on GCE node
W20260927 21:12:28.417111 3644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:28.417732 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:28.417970 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:28.419124 3639 hybrid_clock.cc:648] HybridClock initialized: now 1790543548419109 us; error 34 us; skew 500 ppm
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:28.421554 3639 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:28.421984 3639 webserver.cc:460] Webserver started at http://127.0.98.2:39109/ using document root <none> and password file <none>
I20260927 21:12:28.422169 3639 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:28.422210 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:28.422302 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:28.423338 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "c36d814b2080499ca6ae1f3930dbfc06"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "928b629c6b7a6852f83c0eb7aad4cb71"
server_key_iv: "474e1d83f4c70eccd4986a7d70d94d2b"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.423653 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c36d814b2080499ca6ae1f3930dbfc06"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "928b629c6b7a6852f83c0eb7aad4cb71"
server_key_iv: "474e1d83f4c70eccd4986a7d70d94d2b"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.425071 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:28.425943 3654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.426141 3639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:28.426239 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "c36d814b2080499ca6ae1f3930dbfc06"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "928b629c6b7a6852f83c0eb7aad4cb71"
server_key_iv: "474e1d83f4c70eccd4986a7d70d94d2b"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.426307 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:28.439195 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:28.441447 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:28.441570 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:28.441787 3639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:28.442082 3639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:28.442116 3639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.442137 3639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:28.442159 3639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.447666 3639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46775
I20260927 21:12:28.447741 3767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46775 every 8 connection(s)
I20260927 21:12:28.448050 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:28.457469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 3639
I20260927 21:12:28.457589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:28.457775 392 external_mini_cluster.cc:1428] Setting key b8a148b641504278d216249d80fee15b
I20260927 21:12:28.457795 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:28.448942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46809 (local address 127.0.98.62:46631)
0927 21:12:28.448999 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:28.449001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:28.449201 (+ 200us) server_negotiation.cc:407] Connection header received
0927 21:12:28.449808 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:28.449810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:28.449822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:28.449874 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:28.450872 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.451370 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.452388 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.452578 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.452633 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:28.454956 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:28.454958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:28.454959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:28.454968 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:28.456293 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.456720 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.456721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.456721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.456775 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.457005 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.457006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.457006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.457127 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:28.457183 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:28.457528 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 21:12:28.457702 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:12:28.458128 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:28.448910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46631 (local address 127.0.98.2:46809)
0927 21:12:28.449075 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:28.449112 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:28.449700 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:28.449950 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:28.449955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:28.450561 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:28.450764 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:28.450769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.451467 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.451468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:28.452294 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:28.452297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.452652 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.452653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:28.452777 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:28.453337 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:28.453345 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:28.454860 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:28.456377 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:28.456386 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:28.456389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:28.456651 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:28.456829 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:28.456830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:28.456830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:28.456950 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:28.457182 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:28.457188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:28.457397 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 21:12:28.457500 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:12:28.457762 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260927 21:12:28.458688 3768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46631
I20260927 21:12:28.458792 3768 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:28.458992 3768 heartbeater.cc:511] Master 127.0.98.62:46631 requested a full tablet report, sending...
I20260927 21:12:28.459362 3443 ts_manager.cc:194] Registered new tserver with Master: c36d814b2080499ca6ae1f3930dbfc06 (127.0.98.2:46775)
I20260927 21:12:28.459729 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46809
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:28.485559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46631
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:28.575038 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:28.575281 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:28.575317 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:28.576699 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:28.576757 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:28.576788 3773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:28.578406 3773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:28.578632 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:28.578845 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:28.584079 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:28.584203 3773 server_base.cc:1034] running on GCE node
W20260927 21:12:28.584147 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:28.584147 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:28.584579 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:28.585214 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:28.586495 3773 hybrid_clock.cc:648] HybridClock initialized: now 1790543548586466 us; error 44 us; skew 500 ppm
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:28.589632 3773 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:28.590471 3773 webserver.cc:460] Webserver started at http://127.0.98.3:44939/ using document root <none> and password file <none>
I20260927 21:12:28.590730 3773 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:28.590781 3773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:28.590898 3773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:28.592088 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "d3825f5d116447098199e57a970cf762"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "af0e63c6eb1e54e0d593e16adfe859d9"
server_key_iv: "3b076cb722d4269cb1b7a15b711aa5f1"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.592422 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d3825f5d116447098199e57a970cf762"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "af0e63c6eb1e54e0d593e16adfe859d9"
server_key_iv: "3b076cb722d4269cb1b7a15b711aa5f1"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.593902 3773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:28.595012 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.595237 3773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:28.595364 3773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "d3825f5d116447098199e57a970cf762"
format_stamp: "Formatted at 2026-09-27 21:12:28 on dist-test-slave-r9x1"
server_key: "af0e63c6eb1e54e0d593e16adfe859d9"
server_key_iv: "3b076cb722d4269cb1b7a15b711aa5f1"
server_key_version: "encryptionkey@0"
I20260927 21:12:28.595433 3773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:28.607399 3773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:28.609927 3773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:28.610075 3773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:28.610296 3773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:28.610651 3773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:28.610685 3773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.610714 3773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:28.610738 3773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:28.616482 3773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42595
I20260927 21:12:28.616542 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42595 every 8 connection(s)
I20260927 21:12:28.616844 3773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:28.620838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 3773
I20260927 21:12:28.620919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:28.621078 392 external_mini_cluster.cc:1428] Setting key 852449ecc1347ecaffb9cb40f5c273f3
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:28.627300 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:28.617703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35591 (local address 127.0.98.62:46631)
0927 21:12:28.617758 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:28.617760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:28.617979 (+ 219us) server_negotiation.cc:407] Connection header received
0927 21:12:28.618534 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:28.618535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:28.618546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:28.618607 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:28.619701 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.620196 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.621341 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.621612 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.621678 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:28.624011 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:28.624012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:28.624013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:28.624024 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:28.625790 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.626239 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.626241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.626241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.626299 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.626662 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.626663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.626664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.626790 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:28.626870 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:28.627160 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 21:12:28.627232 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:12:28.627487 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:28.617611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46631 (local address 127.0.98.3:35591)
0927 21:12:28.617834 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:28.617884 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:28.618412 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:28.618706 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:28.618710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:28.619305 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:28.619509 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:28.619514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.620307 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.620308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:28.621222 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:28.621226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.621687 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.621688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:28.621829 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:28.622415 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:28.622424 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:28.623886 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:28.625895 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:28.625904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:28.625906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:28.626149 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:28.626407 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:28.626408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:28.626408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:28.626528 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:28.626873 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:28.626879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:28.627030 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 21:12:28.627123 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:12:28.627259 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260927 21:12:28.627947 3902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46631
I20260927 21:12:28.628037 3902 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:28.628214 3902 heartbeater.cc:511] Master 127.0.98.62:46631 requested a full tablet report, sending...
I20260927 21:12:28.628649 3443 ts_manager.cc:194] Registered new tserver with Master: d3825f5d116447098199e57a970cf762 (127.0.98.3:42595)
I20260927 21:12:28.629002 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35591
I20260927 21:12:28.632872 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543548, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543548, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:28.656497 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:28.648834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.0.98.62:46631)
0927 21:12:28.648895 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:28.648897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:28.648907 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:28.648987 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:28.648988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:28.648999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:28.649045 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:28.649639 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.650120 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.650859 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:28.651038 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:28.651091 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:28.653433 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:28.653435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:28.653435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:28.653444 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:28.655350 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.655714 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.655716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.655716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.655770 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:28.655940 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:28.655940 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:28.655941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:28.656102 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:28.656192 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:28.656310 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:12:28.656394 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 21:12:28.656899 3443 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60434
I20260927 21:12:28.657398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 3505
I20260927 21:12:28.661756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 3639
I20260927 21:12:28.665670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 3773
I20260927 21:12:28.669623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 3411
2026-09-27T21:12:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1809 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3914](info): set up 2 sockets
Sep 27 21:12:28 dist-test-slave-r9x1 krb5kdc[3914](info): commencing operation
krb5kdc: starting...
W20260927 21:12:29.645677 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:29 dist-test-slave-r9x1 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:29.773183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42453
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:29.856734 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:29.856976 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:29.857000 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:29.858369 3927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:29.858428 3927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:29.858443 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:29.858453 3927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:29.858464 3927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:29.860055 3927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:29.860311 3927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:29.860541 3927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:29.865506 3933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:29.865540 3932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:29.865549 3927 server_base.cc:1034] running on GCE node
W20260927 21:12:29.865516 3935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:29.865935 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:29.866235 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:29.867383 3927 hybrid_clock.cc:648] HybridClock initialized: now 1790543549867365 us; error 38 us; skew 500 ppm
Sep 27 21:12:29 dist-test-slave-r9x1 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:29.869839 3927 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:29.870307 3927 webserver.cc:460] Webserver started at http://127.0.98.62:40987/ using document root <none> and password file <none>
I20260927 21:12:29.870508 3927 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:29.870563 3927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:29.870692 3927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:29.871749 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "c1da083bd5854acdbad37802351b3cb5"
format_stamp: "Formatted at 2026-09-27 21:12:29 on dist-test-slave-r9x1"
server_key: "c74eeb7990244b6b0f83e004d167f2fa"
server_key_iv: "161e98cdf06b4ecbf3140c1777c92fd2"
server_key_version: "encryptionkey@0"
I20260927 21:12:29.872094 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1da083bd5854acdbad37802351b3cb5"
format_stamp: "Formatted at 2026-09-27 21:12:29 on dist-test-slave-r9x1"
server_key: "c74eeb7990244b6b0f83e004d167f2fa"
server_key_iv: "161e98cdf06b4ecbf3140c1777c92fd2"
server_key_version: "encryptionkey@0"
I20260927 21:12:29.873692 3927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:29.874981 3942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:29.875285 3927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:29.875397 3927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "c1da083bd5854acdbad37802351b3cb5"
format_stamp: "Formatted at 2026-09-27 21:12:29 on dist-test-slave-r9x1"
server_key: "c74eeb7990244b6b0f83e004d167f2fa"
server_key_iv: "161e98cdf06b4ecbf3140c1777c92fd2"
server_key_version: "encryptionkey@0"
I20260927 21:12:29.875480 3927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:29.888433 3927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:29.890697 3927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:29.890820 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:29.894572 3927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42453
I20260927 21:12:29.894606 3994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42453 every 8 connection(s)
I20260927 21:12:29.894951 3927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:29.895876 3995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:29.898008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 3927
I20260927 21:12:29.898102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:29.898270 392 external_mini_cluster.cc:1428] Setting key ed64c153ba0e614125a9ca2efb4dd8d0
I20260927 21:12:29.898377 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5: Bootstrap starting.
I20260927 21:12:29.899019 3995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:29.899287 3995 log.cc:824] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:29.899900 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5: No bootstrap required, opened a new log
I20260927 21:12:29.901733 3995 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da083bd5854acdbad37802351b3cb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42453 } }
I20260927 21:12:29.901886 3995 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:29.901921 3995 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1da083bd5854acdbad37802351b3cb5, State: Initialized, Role: FOLLOWER
I20260927 21:12:29.902052 3995 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da083bd5854acdbad37802351b3cb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42453 } }
I20260927 21:12:29.902122 3995 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:29.902153 3995 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:29.902199 3995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:29.902923 3995 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da083bd5854acdbad37802351b3cb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42453 } }
I20260927 21:12:29.903034 3995 leader_election.cc:302] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1da083bd5854acdbad37802351b3cb5; no voters:
I20260927 21:12:29.903198 3995 leader_election.cc:288] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:29.903225 4000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:29.903362 4000 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [term 1 LEADER]: Becoming Leader. State: Replica: c1da083bd5854acdbad37802351b3cb5, State: Running, Role: LEADER
I20260927 21:12:29.903435 4000 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da083bd5854acdbad37802351b3cb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42453 } }
I20260927 21:12:29.903467 3995 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:29.903744 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1da083bd5854acdbad37802351b3cb5. Latest consensus state: current_term: 1 leader_uuid: "c1da083bd5854acdbad37802351b3cb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da083bd5854acdbad37802351b3cb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42453 } } }
I20260927 21:12:29.903837 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [sys.catalog]: This master's current role is: LEADER
Sep 27 21:12:29 dist-test-slave-r9x1 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260927 21:12:29.905088 4015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 21:12:29.905551 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1da083bd5854acdbad37802351b3cb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da083bd5854acdbad37802351b3cb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42453 } } }
I20260927 21:12:29.905741 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5 [sys.catalog]: This master's current role is: LEADER
W20260927 21:12:29.905808 4015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:12:29.905880 4005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:29.906055 4005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:29.908124 4005 catalog_manager.cc:1347] Generated new cluster ID: ca4bd46a251a4a7583a5d748a8f4f922
I20260927 21:12:29.908179 4005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:29.909221 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:29.898792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.0.98.62:42453)
0927 21:12:29.898985 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:29.898993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:29.899026 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:12:29.899631 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:29.899639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:29.899692 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:29.899826 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:29.900461 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:29.901431 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:29.902218 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:29.902829 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:29.902954 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:29.905654 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:29.905658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:29.905659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:29.905672 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:29.907452 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:29.907890 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:29.907893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:29.907893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:29.908000 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:29.908222 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:29.908223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:29.908223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:29.908562 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:29.908738 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:29.908843 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:12:29.908992 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260927 21:12:29.942860 4005 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:29.943413 4005 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:29.953138 4005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1da083bd5854acdbad37802351b3cb5: Generated new TSK 0
I20260927 21:12:29.953368 4005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:29.985204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42453
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:30.068280 4021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:30.068542 4021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:30.068567 4021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:30.069880 4021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:30.069932 4021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:30.069962 4021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:30.071558 4021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:30.071801 4021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:30.072032 4021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:30.077183 4029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:30.077195 4027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:30.077229 4021 server_base.cc:1034] running on GCE node
W20260927 21:12:30.077200 4026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:30.077690 4021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:30.077988 4021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:30.079164 4021 hybrid_clock.cc:648] HybridClock initialized: now 1790543550079158 us; error 80 us; skew 500 ppm
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:30.082095 4021 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:30.082582 4021 webserver.cc:460] Webserver started at http://127.0.98.1:35641/ using document root <none> and password file <none>
I20260927 21:12:30.082785 4021 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:30.082834 4021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:30.082954 4021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:30.083998 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "89588255e43d41aeac306845b39f64c1"
format_stamp: "Formatted at 2026-09-27 21:12:30 on dist-test-slave-r9x1"
server_key: "dbc25f7bfff43f340f129ebb7bf5ad86"
server_key_iv: "8183dd6252d64c6c334b8d402c347621"
server_key_version: "encryptionkey@0"
I20260927 21:12:30.084336 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "89588255e43d41aeac306845b39f64c1"
format_stamp: "Formatted at 2026-09-27 21:12:30 on dist-test-slave-r9x1"
server_key: "dbc25f7bfff43f340f129ebb7bf5ad86"
server_key_iv: "8183dd6252d64c6c334b8d402c347621"
server_key_version: "encryptionkey@0"
I20260927 21:12:30.085711 4021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:30.086573 4036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:30.086828 4021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:30.086917 4021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "89588255e43d41aeac306845b39f64c1"
format_stamp: "Formatted at 2026-09-27 21:12:30 on dist-test-slave-r9x1"
server_key: "dbc25f7bfff43f340f129ebb7bf5ad86"
server_key_iv: "8183dd6252d64c6c334b8d402c347621"
server_key_version: "encryptionkey@0"
I20260927 21:12:30.086995 4021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:30.116863 4021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:30.119004 4021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:30.119132 4021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:30.119369 4021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:30.119673 4021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:30.119705 4021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:30.119728 4021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:30.119750 4021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:30.125445 4021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39821
I20260927 21:12:30.125495 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39821 every 8 connection(s)
I20260927 21:12:30.125819 4021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:12:30.130414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 4021
I20260927 21:12:30.130518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:30.130681 392 external_mini_cluster.cc:1428] Setting key f1e87551d5de151e2538b49151df87ac
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:30.135932 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:30.126542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59193 (local address 127.0.98.62:42453)
0927 21:12:30.126608 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:30.126610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:30.126804 (+ 194us) server_negotiation.cc:407] Connection header received
0927 21:12:30.127918 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:30.127919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:30.127930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:30.127986 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:30.128640 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.129134 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.130269 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.130494 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.130574 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:30.133111 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:30.133113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:30.133113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:30.133122 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:30.134569 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.134958 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.134959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.134959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.135027 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.135264 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.135265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.135265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.135409 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:30.135478 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:30.135720 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 21:12:30.135828 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:12:30.135977 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:30.126500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42453 (local address 127.0.98.1:59193)
0927 21:12:30.126675 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:30.126720 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:30.127806 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:30.128063 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:30.128066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:30.128344 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:30.128571 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:30.128575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.129232 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.129233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:30.130148 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:30.130151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.130600 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.130601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:30.130759 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:30.131331 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:30.131344 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:30.132994 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:30.134660 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:30.134667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:30.134669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:30.134891 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:30.135097 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:30.135098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:30.135098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:30.135205 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:30.135467 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:30.135470 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:30.135598 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 21:12:30.135676 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:12:30.135806 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260927 21:12:30.136471 4150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42453
I20260927 21:12:30.136590 4150 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:30.136806 4150 heartbeater.cc:511] Master 127.0.98.62:42453 requested a full tablet report, sending...
I20260927 21:12:30.137399 3959 ts_manager.cc:194] Registered new tserver with Master: 89588255e43d41aeac306845b39f64c1 (127.0.98.1:39821)
I20260927 21:12:30.138003 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59193
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:30.158105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42453
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:30.239135 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:30.239351 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:30.239383 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:30.240592 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:30.240645 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:30.240672 4155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:30.242228 4155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:30.242471 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:30.242692 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:12:30.247711 4155 server_base.cc:1034] running on GCE node
W20260927 21:12:30.247615 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:30.247730 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:30.247684 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:30.248209 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:30.248452 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:30.249601 4155 hybrid_clock.cc:648] HybridClock initialized: now 1790543550249596 us; error 39 us; skew 500 ppm
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:30.252262 4155 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:30.252794 4155 webserver.cc:460] Webserver started at http://127.0.98.2:38629/ using document root <none> and password file <none>
I20260927 21:12:30.253002 4155 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:30.253047 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:30.253157 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:30.254276 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "f093758bff4b4feba8f6e914396100fd"
format_stamp: "Formatted at 2026-09-27 21:12:30 on dist-test-slave-r9x1"
server_key: "e6392062fb632f086c5b2199cefd948e"
server_key_iv: "f007018a7e682a1a23d3821158ebfdc0"
server_key_version: "encryptionkey@0"
I20260927 21:12:30.254623 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f093758bff4b4feba8f6e914396100fd"
format_stamp: "Formatted at 2026-09-27 21:12:30 on dist-test-slave-r9x1"
server_key: "e6392062fb632f086c5b2199cefd948e"
server_key_iv: "f007018a7e682a1a23d3821158ebfdc0"
server_key_version: "encryptionkey@0"
I20260927 21:12:30.255975 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:30.256752 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:30.256947 4155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:30.257037 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "f093758bff4b4feba8f6e914396100fd"
format_stamp: "Formatted at 2026-09-27 21:12:30 on dist-test-slave-r9x1"
server_key: "e6392062fb632f086c5b2199cefd948e"
server_key_iv: "f007018a7e682a1a23d3821158ebfdc0"
server_key_version: "encryptionkey@0"
I20260927 21:12:30.257102 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:30.265888 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:30.268098 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:30.268221 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:30.268441 4155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:30.268735 4155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:30.268774 4155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:30.268797 4155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:30.268808 4155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:30.274608 4155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39693
I20260927 21:12:30.274698 4283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39693 every 8 connection(s)
I20260927 21:12:30.275085 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:30.283164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 4155
I20260927 21:12:30.283270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:30.283422 392 external_mini_cluster.cc:1428] Setting key cc130a48d149052246710bb3e4d7bea4
I20260927 21:12:30.286698 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:30.275907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44409 (local address 127.0.98.62:42453)
0927 21:12:30.275982 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:30.275984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:30.276169 (+ 185us) server_negotiation.cc:407] Connection header received
0927 21:12:30.277303 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:30.277305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:30.277320 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:30.277437 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:30.278207 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.278732 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.279947 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.280168 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.280241 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:30.282514 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:30.282516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:30.282516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:30.282525 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:30.284196 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.284835 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.284837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.284837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.284907 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.285554 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.285555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.285556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.285777 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:30.285876 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:30.286527 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 21:12:30.286628 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:12:30.287012 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:30.275854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42453 (local address 127.0.98.2:44409)
0927 21:12:30.276032 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:30.276073 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:30.277146 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:30.277504 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:30.277508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:30.277826 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:30.278100 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:30.278105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.278839 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.278841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:30.279853 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:30.279856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.280292 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.280293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:30.280430 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:30.280963 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:30.280972 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:30.282408 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:30.284449 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:30.284461 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:30.284464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:30.284743 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:30.285078 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:30.285080 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:30.285081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:30.285442 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:30.285912 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:30.285926 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:30.286220 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 21:12:30.286488 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 21:12:30.286750 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260927 21:12:30.287621 4284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42453
I20260927 21:12:30.287726 4284 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:30.287922 4284 heartbeater.cc:511] Master 127.0.98.62:42453 requested a full tablet report, sending...
I20260927 21:12:30.288362 3959 ts_manager.cc:194] Registered new tserver with Master: f093758bff4b4feba8f6e914396100fd (127.0.98.2:39693)
I20260927 21:12:30.288738 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44409
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:30.312767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42453
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:30.403501 4289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:30.403738 4289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:30.403822 4289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:30.405467 4289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:30.405530 4289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:30.405571 4289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:30.407536 4289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:30.407858 4289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:30.408118 4289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:30.413730 4295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:30.413754 4289 server_base.cc:1034] running on GCE node
W20260927 21:12:30.413733 4297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:30.413730 4294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:30.414147 4289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:30.414391 4289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:30.415566 4289 hybrid_clock.cc:648] HybridClock initialized: now 1790543550415552 us; error 29 us; skew 500 ppm
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:30.418202 4289 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:30.418700 4289 webserver.cc:460] Webserver started at http://127.0.98.3:38185/ using document root <none> and password file <none>
I20260927 21:12:30.418931 4289 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:30.418982 4289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:30.419111 4289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:30.420161 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "71f565d17b774edaa5fd997978bc33d1"
format_stamp: "Formatted at 2026-09-27 21:12:30 on dist-test-slave-r9x1"
server_key: "75044a38ab56647cbef6c6ab58996f36"
server_key_iv: "d2bc585f69053ebcf153b86cfa5e1f2d"
server_key_version: "encryptionkey@0"
I20260927 21:12:30.420512 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "71f565d17b774edaa5fd997978bc33d1"
format_stamp: "Formatted at 2026-09-27 21:12:30 on dist-test-slave-r9x1"
server_key: "75044a38ab56647cbef6c6ab58996f36"
server_key_iv: "d2bc585f69053ebcf153b86cfa5e1f2d"
server_key_version: "encryptionkey@0"
I20260927 21:12:30.421940 4289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:30.422775 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:30.422996 4289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:30.423102 4289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "71f565d17b774edaa5fd997978bc33d1"
format_stamp: "Formatted at 2026-09-27 21:12:30 on dist-test-slave-r9x1"
server_key: "75044a38ab56647cbef6c6ab58996f36"
server_key_iv: "d2bc585f69053ebcf153b86cfa5e1f2d"
server_key_version: "encryptionkey@0"
I20260927 21:12:30.423177 4289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:30.441797 4289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:30.444418 4289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:30.444561 4289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:30.444802 4289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:30.445122 4289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:30.445156 4289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:30.445186 4289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:30.445211 4289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:30.451043 4289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34345
I20260927 21:12:30.451086 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34345 every 8 connection(s)
I20260927 21:12:30.451432 4289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:30.458688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 4289
I20260927 21:12:30.458787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:30.458978 392 external_mini_cluster.cc:1428] Setting key 5f2e6012817c4e5694dcec8172b3451c
I20260927 21:12:30.461477 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:30.452172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59431 (local address 127.0.98.62:42453)
0927 21:12:30.452235 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:30.452237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:30.452460 (+ 223us) server_negotiation.cc:407] Connection header received
0927 21:12:30.453487 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:30.453489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:30.453500 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:30.453579 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:30.454274 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.454789 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.455919 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.456130 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.456193 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:30.458501 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:30.458503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:30.458503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:30.458513 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:30.460040 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.460517 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.460518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.460518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.460577 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.460794 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.460795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.460795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.460926 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:30.461010 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:30.461239 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 21:12:30.461349 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:12:30.461632 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:30.452144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42453 (local address 127.0.98.3:59431)
0927 21:12:30.452316 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:30.452377 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:30.453391 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:30.453662 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:30.453666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:30.453959 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:30.454165 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:30.454170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.454900 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.454901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:30.455808 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:30.455811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.456220 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.456221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:30.456349 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:30.456868 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:30.456878 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:30.458382 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:30.460138 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:30.460147 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:30.460149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:30.460451 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:30.460621 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:30.460621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:30.460622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:30.460721 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:30.461005 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:30.461007 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:30.461137 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 21:12:30.461219 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:12:30.461395 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260927 21:12:30.462067 4418 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42453
I20260927 21:12:30.462157 4418 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:30.462353 4418 heartbeater.cc:511] Master 127.0.98.62:42453 requested a full tablet report, sending...
I20260927 21:12:30.462675 3959 ts_manager.cc:194] Registered new tserver with Master: 71f565d17b774edaa5fd997978bc33d1 (127.0.98.3:34345)
I20260927 21:12:30.463052 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59431
I20260927 21:12:30.471486 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:12:30.495297 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:30.486971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.0.98.1:39821)
0927 21:12:30.487138 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:30.487143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:30.487152 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:30.487176 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:30.487178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:30.487191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:30.487259 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:30.487842 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.488608 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.489377 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.489708 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.489769 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:30.492220 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:30.492230 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:30.492231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:30.492240 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:30.493993 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.494347 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.494351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.494351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.494412 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.494589 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.494590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.494591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.494763 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:30.494894 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:30.495075 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 21:12:30.495175 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:30.503916 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:30.496727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.0.98.62:42453)
0927 21:12:30.496783 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:30.496785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:30.496798 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:12:30.496854 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:30.496855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:30.496865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:30.496910 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:30.497389 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.497890 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.498576 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:30.498765 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:30.498826 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:30.501323 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:30.501324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:30.501324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:30.501333 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:30.502686 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.503037 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.503039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.503039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.503093 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:30.503256 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:30.503257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:30.503257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:30.503452 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:30.503546 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:30.503678 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 21:12:30.503762 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 21:12:30.504303 3959 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52778
I20260927 21:12:30.504720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 4021
I20260927 21:12:30.509110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 4155
I20260927 21:12:30.513612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 4289
I20260927 21:12:30.518509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 3927
2026-09-27T21:12:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1851 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[4430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[4430](info): set up 2 sockets
Sep 27 21:12:30 dist-test-slave-r9x1 krb5kdc[4430](info): commencing operation
krb5kdc: starting...
W20260927 21:12:31.493053 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:31 dist-test-slave-r9x1 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:31.613646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44839
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:31.700898 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:31.701122 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:31.701146 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:31.702355 4443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:31.702409 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:31.702421 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:31.702440 4443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:31.702451 4443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:31.704026 4443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:31.704258 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:31.704450 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:31.709370 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:31.709380 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:31.709410 4443 server_base.cc:1034] running on GCE node
W20260927 21:12:31.709393 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:31.709939 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:31.710249 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:31.711412 4443 hybrid_clock.cc:648] HybridClock initialized: now 1790543551711390 us; error 40 us; skew 500 ppm
Sep 27 21:12:31 dist-test-slave-r9x1 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:31.713930 4443 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:31.714311 4443 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:31.714368 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:31.714490 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:31.715605 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "92983374afa64e2593023ffd0b45cc1f"
format_stamp: "Formatted at 2026-09-27 21:12:31 on dist-test-slave-r9x1"
server_key: "9546d1791a729c370328a60a7864dd19"
server_key_iv: "3a77f8684a692214f5d634cdba4872a9"
server_key_version: "encryptionkey@0"
I20260927 21:12:31.715947 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "92983374afa64e2593023ffd0b45cc1f"
format_stamp: "Formatted at 2026-09-27 21:12:31 on dist-test-slave-r9x1"
server_key: "9546d1791a729c370328a60a7864dd19"
server_key_iv: "3a77f8684a692214f5d634cdba4872a9"
server_key_version: "encryptionkey@0"
I20260927 21:12:31.717896 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 21:12:31.718815 4457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:31.718991 4443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:31.719097 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "92983374afa64e2593023ffd0b45cc1f"
format_stamp: "Formatted at 2026-09-27 21:12:31 on dist-test-slave-r9x1"
server_key: "9546d1791a729c370328a60a7864dd19"
server_key_iv: "3a77f8684a692214f5d634cdba4872a9"
server_key_version: "encryptionkey@0"
I20260927 21:12:31.719172 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:31.741189 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:31.743357 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:31.743505 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:31.747422 4443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44839
I20260927 21:12:31.747470 4509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44839 every 8 connection(s)
I20260927 21:12:31.747753 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:31.748725 4510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:31.749473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 4443
I20260927 21:12:31.749576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:31.749797 392 external_mini_cluster.cc:1428] Setting key bf6cfb533058b61d29028c20524ef733
I20260927 21:12:31.751878 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f: Bootstrap starting.
I20260927 21:12:31.752527 4510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:31.752794 4510 log.cc:824] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:31.753500 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f: No bootstrap required, opened a new log
I20260927 21:12:31.755614 4510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92983374afa64e2593023ffd0b45cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } }
I20260927 21:12:31.755766 4510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:31.755800 4510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92983374afa64e2593023ffd0b45cc1f, State: Initialized, Role: FOLLOWER
I20260927 21:12:31.755934 4510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92983374afa64e2593023ffd0b45cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } }
I20260927 21:12:31.756026 4510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:31.756064 4510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:31.756111 4510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [term 0 FOLLOWER]: Advancing to term 1
Sep 27 21:12:31 dist-test-slave-r9x1 krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:31.756840 4510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92983374afa64e2593023ffd0b45cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } }
I20260927 21:12:31.756937 4510 leader_election.cc:302] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92983374afa64e2593023ffd0b45cc1f; no voters:
I20260927 21:12:31.757112 4510 leader_election.cc:288] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:31.757180 4515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:31.757347 4515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [term 1 LEADER]: Becoming Leader. State: Replica: 92983374afa64e2593023ffd0b45cc1f, State: Running, Role: LEADER
I20260927 21:12:31.757457 4510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:31.757486 4515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92983374afa64e2593023ffd0b45cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } }
I20260927 21:12:31.757979 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92983374afa64e2593023ffd0b45cc1f. Latest consensus state: current_term: 1 leader_uuid: "92983374afa64e2593023ffd0b45cc1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92983374afa64e2593023ffd0b45cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } } }
I20260927 21:12:31.758133 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:31.758133 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92983374afa64e2593023ffd0b45cc1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92983374afa64e2593023ffd0b45cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44839 } } }
I20260927 21:12:31.758287 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:31.758443 4524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:31.758966 4524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:31.761503 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:31.750379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.0.98.62:44839)
0927 21:12:31.750563 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:31.750573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:31.750688 (+ 115us) server_negotiation.cc:407] Connection header received
0927 21:12:31.751167 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:31.751175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:31.751231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:31.751355 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:31.752012 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:31.753038 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:31.753884 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:31.754426 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:31.754551 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:31.757201 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:31.757206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:31.757206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:31.757220 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:31.759847 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:31.760293 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:31.760296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:31.760297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:31.760400 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:31.760604 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:31.760605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:31.760606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:31.760865 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:31.760998 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:31.761118 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 21:12:31.761260 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260927 21:12:31.765625 4524 catalog_manager.cc:1347] Generated new cluster ID: e9120e2457074a2bae340cbb057f61a9
I20260927 21:12:31.765691 4524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:31.789039 4524 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:31.789657 4524 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:31.794148 4524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f: Generated new TSK 0
I20260927 21:12:31.794358 4524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:31.829845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44839
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:31.913017 4536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:31.913256 4536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:31.913316 4536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:31.914616 4536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:31.914665 4536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:31.914693 4536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:31.916559 4536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:31.916828 4536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:31.917073 4536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:31.922258 4542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:31.922415 4536 server_base.cc:1034] running on GCE node
W20260927 21:12:31.922276 4544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:31.922276 4541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:31.922713 4536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:31.922994 4536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:31.924158 4536 hybrid_clock.cc:648] HybridClock initialized: now 1790543551924131 us; error 45 us; skew 500 ppm
Sep 27 21:12:31 dist-test-slave-r9x1 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:31.931058 4536 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:31.931427 4536 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:31.931476 4536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:31.931587 4536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:31.932679 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "13e834b033344897a353bf5cc0e1d226"
format_stamp: "Formatted at 2026-09-27 21:12:31 on dist-test-slave-r9x1"
server_key: "99a1012dd68c78261b492a6555caf100"
server_key_iv: "6776744a0b613c16b9201c2fd5dc2206"
server_key_version: "encryptionkey@0"
I20260927 21:12:31.933017 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "13e834b033344897a353bf5cc0e1d226"
format_stamp: "Formatted at 2026-09-27 21:12:31 on dist-test-slave-r9x1"
server_key: "99a1012dd68c78261b492a6555caf100"
server_key_iv: "6776744a0b613c16b9201c2fd5dc2206"
server_key_version: "encryptionkey@0"
I20260927 21:12:31.934384 4536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:31.935223 4550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:31.935397 4536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:12:31.935477 4536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "13e834b033344897a353bf5cc0e1d226"
format_stamp: "Formatted at 2026-09-27 21:12:31 on dist-test-slave-r9x1"
server_key: "99a1012dd68c78261b492a6555caf100"
server_key_iv: "6776744a0b613c16b9201c2fd5dc2206"
server_key_version: "encryptionkey@0"
I20260927 21:12:31.935549 4536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:31.950805 4536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:31.953009 4536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:31.953136 4536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:31.953401 4536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:31.953744 4536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:31.953780 4536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:31.953809 4536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:31.953835 4536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:31.959707 4536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36721
I20260927 21:12:31.959769 4663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36721 every 8 connection(s)
I20260927 21:12:31.960019 4536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:12:31.965451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 4536
I20260927 21:12:31.965564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:31.965741 392 external_mini_cluster.cc:1428] Setting key b38b2b07fca6520c3163004f7fe0db2a
Sep 27 21:12:31 dist-test-slave-r9x1 krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:31.969801 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:31.960790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42259 (local address 127.0.98.62:44839)
0927 21:12:31.960857 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:31.960859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:31.961035 (+ 176us) server_negotiation.cc:407] Connection header received
0927 21:12:31.961599 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:31.961600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:31.961610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:31.961652 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:31.962649 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:31.963175 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:31.964276 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:31.964485 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:31.964548 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:31.966971 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:31.966973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:31.966973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:31.966982 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:31.968241 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:31.968640 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:31.968641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:31.968641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:31.968709 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:31.969000 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:31.969000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:31.969001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:31.969143 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:31.969220 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:31.969589 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 21:12:31.969696 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:12:31.969929 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:31.960747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44839 (local address 127.0.98.1:42259)
0927 21:12:31.960911 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:31.960956 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:31.961490 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:31.961725 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:31.961728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:31.962006 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:31.962562 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:31.962568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:31.963275 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:31.963277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:31.964160 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:31.964163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:31.964592 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:31.964593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:31.964714 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:31.965222 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:31.965232 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:31.966873 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:31.968318 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:31.968325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:31.968327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:31.968569 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:31.968798 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:31.968799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:31.968799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:31.968919 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:31.969252 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:31.969258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:31.969444 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 21:12:31.969550 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:12:31.969726 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260927 21:12:31.970392 4664 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44839
I20260927 21:12:31.970484 4664 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:31.970645 4664 heartbeater.cc:511] Master 127.0.98.62:44839 requested a full tablet report, sending...
I20260927 21:12:31.971115 4474 ts_manager.cc:194] Registered new tserver with Master: 13e834b033344897a353bf5cc0e1d226 (127.0.98.1:36721)
I20260927 21:12:31.971711 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42259
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:31.992851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44839
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:32.074817 4669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:32.075062 4669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:32.075098 4669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:32.076570 4669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:32.076643 4669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:32.076687 4669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:32.078437 4669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:32.078776 4669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:32.079010 4669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:32.083887 4677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.083925 4675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:32.083961 4669 server_base.cc:1034] running on GCE node
W20260927 21:12:32.083941 4674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:32.084388 4669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:32.084668 4669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:32.085835 4669 hybrid_clock.cc:648] HybridClock initialized: now 1790543552085814 us; error 38 us; skew 500 ppm
Sep 27 21:12:32 dist-test-slave-r9x1 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:32.088459 4669 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:32.088811 4669 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:32.088863 4669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:32.088999 4669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:32.090344 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "273df4d30eda449494b5a9ea9ff030b8"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "8f7c1439248401cafe05b21ffaea4a28"
server_key_iv: "de82dafdc172ef77ae63e29e2591510a"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.090850 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "273df4d30eda449494b5a9ea9ff030b8"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "8f7c1439248401cafe05b21ffaea4a28"
server_key_iv: "de82dafdc172ef77ae63e29e2591510a"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.092203 4669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:32.092903 4683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.093080 4669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:12:32.093168 4669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "273df4d30eda449494b5a9ea9ff030b8"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "8f7c1439248401cafe05b21ffaea4a28"
server_key_iv: "de82dafdc172ef77ae63e29e2591510a"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.093230 4669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:32.115190 4669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:32.117412 4669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:32.117573 4669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:32.117815 4669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:32.118153 4669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:32.118199 4669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.118229 4669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:32.118247 4669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.124078 4669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43169
I20260927 21:12:32.124109 4796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43169 every 8 connection(s)
I20260927 21:12:32.124405 4669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:12:32.127995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 4669
I20260927 21:12:32.128090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:32.128271 392 external_mini_cluster.cc:1428] Setting key a5563e130eae2be0d42f9835d0c06002
Sep 27 21:12:32 dist-test-slave-r9x1 krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:32.134503 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.125152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42055 (local address 127.0.98.62:44839)
0927 21:12:32.125249 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.125251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.125412 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:12:32.125925 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.125927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.125939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.125995 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.126930 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.127415 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.128580 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.128847 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.128914 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.131310 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.131312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.131312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:32.131321 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.132941 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:32.133450 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:32.133452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:32.133453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:32.133518 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:32.133811 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:32.133812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:32.133812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:32.133961 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.134030 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.134306 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.134405 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:12:32.134649 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.125057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44839 (local address 127.0.98.2:42055)
0927 21:12:32.125259 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.125329 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.125824 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.126070 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.126074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.126337 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:32.126830 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.126835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.127514 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.127515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.128463 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.128467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.128959 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.128960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.129122 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.129695 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:32.129704 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:32.131204 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:32.133050 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:32.133057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:32.133059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:32.133370 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:32.133607 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:32.133608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:32.133609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:32.133735 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:32.134024 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:32.134028 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:32.134171 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.134254 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.134391 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260927 21:12:32.135131 4797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44839
I20260927 21:12:32.135248 4797 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:32.135417 4797 heartbeater.cc:511] Master 127.0.98.62:44839 requested a full tablet report, sending...
I20260927 21:12:32.135793 4474 ts_manager.cc:194] Registered new tserver with Master: 273df4d30eda449494b5a9ea9ff030b8 (127.0.98.2:43169)
I20260927 21:12:32.136161 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42055
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:32.158212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44839
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:32.241103 4802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:32.241380 4802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:32.241411 4802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:32.242712 4802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:32.242758 4802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:32.242785 4802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:32.244386 4802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:32.244632 4802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:32.244833 4802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:32.249883 4810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.249894 4808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.249894 4807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:32.250267 4802 server_base.cc:1034] running on GCE node
I20260927 21:12:32.250497 4802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:32.250739 4802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:32.251917 4802 hybrid_clock.cc:648] HybridClock initialized: now 1790543552251911 us; error 52 us; skew 500 ppm
Sep 27 21:12:32 dist-test-slave-r9x1 krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:32.254662 4802 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:32.255162 4802 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:32.255221 4802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:32.255334 4802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:32.256510 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "0e0f36da9ca24fde8d95e0192bf600f0"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "2b4ef3ae4ceaefc3fc9b7576feddcbe0"
server_key_iv: "d78a1ed93d4cdb51561f475b7d149e79"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.256851 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0e0f36da9ca24fde8d95e0192bf600f0"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "2b4ef3ae4ceaefc3fc9b7576feddcbe0"
server_key_iv: "d78a1ed93d4cdb51561f475b7d149e79"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.258402 4802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:32.259204 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.259382 4802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:32.259457 4802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "0e0f36da9ca24fde8d95e0192bf600f0"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "2b4ef3ae4ceaefc3fc9b7576feddcbe0"
server_key_iv: "d78a1ed93d4cdb51561f475b7d149e79"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.259533 4802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:32.267565 4802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:32.269677 4802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:32.269788 4802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:32.270010 4802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:32.270310 4802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:32.270344 4802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.270365 4802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:32.270376 4802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.275918 4802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42781
I20260927 21:12:32.275982 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42781 every 8 connection(s)
I20260927 21:12:32.276216 4802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:12:32 dist-test-slave-r9x1 krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:32.283141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 4802
I20260927 21:12:32.283265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:32.283445 392 external_mini_cluster.cc:1428] Setting key 0164d98466c0c5e9d6b15f5cd4f7e1ca
I20260927 21:12:32.285506 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.276874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60005 (local address 127.0.98.62:44839)
0927 21:12:32.276930 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.276933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.277108 (+ 175us) server_negotiation.cc:407] Connection header received
0927 21:12:32.277672 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.277673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.277686 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.277735 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.278677 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.279185 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.280160 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.280360 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.280417 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.282596 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.282597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.282598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:32.282607 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.284029 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:32.284441 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:32.284443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:32.284443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:32.284497 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:32.284770 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:32.284772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:32.284772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:32.284909 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.284991 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.285357 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.285439 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:12:32.285682 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.276846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44839 (local address 127.0.98.3:60005)
0927 21:12:32.276986 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.277022 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.277567 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.277818 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.277822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.278086 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:32.278582 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.278587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.279271 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.279272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.280076 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.280078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.280430 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.280431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.280548 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.281068 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:32.281077 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:32.282500 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:32.284114 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:32.284122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:32.284124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:32.284376 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:32.284553 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:32.284554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:32.284554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:32.284688 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:32.285018 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:32.285024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:32.285157 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.285259 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.285462 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260927 21:12:32.286227 4930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44839
I20260927 21:12:32.286335 4930 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:32.286496 4930 heartbeater.cc:511] Master 127.0.98.62:44839 requested a full tablet report, sending...
I20260927 21:12:32.286912 4474 ts_manager.cc:194] Registered new tserver with Master: 0e0f36da9ca24fde8d95e0192bf600f0 (127.0.98.3:42781)
I20260927 21:12:32.287286 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60005
I20260927 21:12:32.295389 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:12:32.302569 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.296682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.0.98.62:44839)
0927 21:12:32.296742 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.296745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.296826 (+ 81us) server_negotiation.cc:407] Connection header received
0927 21:12:32.296888 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.296889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.296899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.296945 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.297541 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.298038 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.298760 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.298952 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.299013 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.300125 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.300127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.300127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:32.300135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.301482 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:32.301826 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:32.301827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:32.301828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:32.301882 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:32.302041 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:32.302042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:32.302042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:32.302199 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.302291 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.302397 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.302475 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:12:32.303982 4474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:32.305783 4474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:32.312800 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.308755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.0.98.2:43169)
0927 21:12:32.308908 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.308911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.309019 (+ 108us) server_negotiation.cc:407] Connection header received
0927 21:12:32.309354 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.309357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.309372 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.309441 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:32.309563 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.310601 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.311588 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.312326 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.312404 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.312468 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.312655 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.312703 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260927 21:12:32.312788 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.308495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43169 (local address 127.0.0.1:60878)
0927 21:12:32.308954 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.308964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.309026 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.309444 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.309445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.309447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:32.309482 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.309483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.310683 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.310684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.311490 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.311492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.312427 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.312429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.312532 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.312550 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.312624 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.312689 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":66,"threads_started":1}
I20260927 21:12:32.312914 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.308465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36721 (local address 127.0.0.1:40924)
0927 21:12:32.308623 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.308672 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.308743 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.309138 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.309142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.309145 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:32.309224 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.309227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.310605 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.310606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.311859 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.311861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.312692 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.312693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.312765 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.312769 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.312799 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.312846 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":79,"thread_start_us":78,"threads_started":1}
I20260927 21:12:32.312796 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.308495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42781 (local address 127.0.0.1:37598)
0927 21:12:32.309107 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.309114 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.309152 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.309374 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.309375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.309377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:32.309430 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.309432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.310439 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.310440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.311804 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.311807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.312607 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.312607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.312663 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.312666 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.312706 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.312744 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260927 21:12:32.312955 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.308786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.0.98.3:42781)
0927 21:12:32.308988 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.308992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.309202 (+ 210us) server_negotiation.cc:407] Connection header received
0927 21:12:32.309244 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.309246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.309260 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.309348 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:32.309506 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.310311 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.311911 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.312448 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.312517 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.312574 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.312834 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.312877 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260927 21:12:32.313376 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.308676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.0.98.1:36721)
0927 21:12:32.308858 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.308864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.308873 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:32.308911 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.308913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.308929 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.309066 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:32.309358 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.310504 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.311921 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.312597 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.313091 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.313175 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.313201 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.313248 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260927 21:12:32.313751 4731 tablet_service.cc:1467] Processing CreateTablet for tablet 10e10ca764eb43d3b175e99620e4cfa2 (DEFAULT_TABLE table=test-table [id=ab04e69da333410da04508e885759892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:32.313747 4864 tablet_service.cc:1467] Processing CreateTablet for tablet 10e10ca764eb43d3b175e99620e4cfa2 (DEFAULT_TABLE table=test-table [id=ab04e69da333410da04508e885759892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:32.314100 4731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e10ca764eb43d3b175e99620e4cfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:32.314108 4864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e10ca764eb43d3b175e99620e4cfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:32.314047 4598 tablet_service.cc:1467] Processing CreateTablet for tablet 10e10ca764eb43d3b175e99620e4cfa2 (DEFAULT_TABLE table=test-table [id=ab04e69da333410da04508e885759892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:32.314352 4598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e10ca764eb43d3b175e99620e4cfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:32.316715 4947 tablet_bootstrap.cc:492] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8: Bootstrap starting.
I20260927 21:12:32.317608 4949 tablet_bootstrap.cc:492] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226: Bootstrap starting.
I20260927 21:12:32.317828 4947 tablet_bootstrap.cc:654] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:32.318243 4949 tablet_bootstrap.cc:654] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:32.318444 4947 log.cc:824] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:32.318799 4949 log.cc:824] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:32.319133 4948 tablet_bootstrap.cc:492] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: Bootstrap starting.
I20260927 21:12:32.319273 4947 tablet_bootstrap.cc:492] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8: No bootstrap required, opened a new log
I20260927 21:12:32.319355 4947 ts_tablet_manager.cc:1397] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:12:32.319509 4949 tablet_bootstrap.cc:492] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226: No bootstrap required, opened a new log
I20260927 21:12:32.319576 4949 ts_tablet_manager.cc:1397] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:32.319730 4948 tablet_bootstrap.cc:654] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:32.320186 4948 log.cc:824] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:32.320987 4948 tablet_bootstrap.cc:492] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: No bootstrap required, opened a new log
I20260927 21:12:32.321048 4948 ts_tablet_manager.cc:1397] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:32.321445 4947 raft_consensus.cc:348] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } }
I20260927 21:12:32.321533 4949 raft_consensus.cc:348] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } }
I20260927 21:12:32.321640 4947 raft_consensus.cc:374] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:32.321668 4949 raft_consensus.cc:374] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:32.321700 4949 raft_consensus.cc:729] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e834b033344897a353bf5cc0e1d226, State: Initialized, Role: FOLLOWER
I20260927 21:12:32.321717 4947 raft_consensus.cc:729] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273df4d30eda449494b5a9ea9ff030b8, State: Initialized, Role: FOLLOWER
I20260927 21:12:32.321969 4949 consensus_queue.cc:260] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } }
I20260927 21:12:32.321978 4947 consensus_queue.cc:260] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } }
I20260927 21:12:32.322302 4947 ts_tablet_manager.cc:1428] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:12:32.322301 4949 ts_tablet_manager.cc:1428] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:12:32.322402 4797 heartbeater.cc:503] Master 127.0.98.62:44839 was elected leader, sending a full tablet report...
I20260927 21:12:32.322757 4664 heartbeater.cc:503] Master 127.0.98.62:44839 was elected leader, sending a full tablet report...
I20260927 21:12:32.323248 4948 raft_consensus.cc:348] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } }
I20260927 21:12:32.323448 4948 raft_consensus.cc:374] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:32.323514 4948 raft_consensus.cc:729] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0f36da9ca24fde8d95e0192bf600f0, State: Initialized, Role: FOLLOWER
I20260927 21:12:32.323693 4948 consensus_queue.cc:260] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } }
I20260927 21:12:32.324015 4948 ts_tablet_manager.cc:1428] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:12:32.324023 4930 heartbeater.cc:503] Master 127.0.98.62:44839 was elected leader, sending a full tablet report...
I20260927 21:12:32.358083 4955 raft_consensus.cc:482] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:32.358234 4955 raft_consensus.cc:504] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } }
I20260927 21:12:32.358587 4955 leader_election.cc:288] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13e834b033344897a353bf5cc0e1d226 (127.0.98.1:36721), 273df4d30eda449494b5a9ea9ff030b8 (127.0.98.2:43169)
I20260927 21:12:32.361680 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.358787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43169 (local address 127.0.98.3:44349)
0927 21:12:32.358825 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.358837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.358879 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.359107 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.359108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.359110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:32.359171 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.359173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.359901 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.359902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.360760 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.360763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.361400 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.361401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.361488 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.361493 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.361552 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.361599 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 21:12:32.361680 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.358897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44349 (local address 127.0.98.2:43169)
0927 21:12:32.358960 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.358962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.358972 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:32.358998 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.358999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.359009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.359091 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:32.359244 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.359820 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.360873 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.361330 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.361384 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.361394 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.361573 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.361610 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:12:32.361814 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.358896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43391 (local address 127.0.98.1:36721)
0927 21:12:32.358958 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.358960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.359272 (+ 312us) server_negotiation.cc:407] Connection header received
0927 21:12:32.359318 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.359319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.359330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.359369 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:32.359533 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.360025 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.361047 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.361498 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.361565 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.361575 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.361715 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.361756 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:12:32.361876 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.358787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36721 (local address 127.0.98.3:43391)
0927 21:12:32.359198 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.359208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.359267 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.359390 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.359391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.359393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:32.359474 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.359476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.360127 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.360128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.360940 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:32.360942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.361577 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.361578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:32.361645 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.361651 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.361705 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.361742 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":128,"thread_start_us":144,"threads_started":1}
I20260927 21:12:32.362212 4618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e10ca764eb43d3b175e99620e4cfa2" candidate_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e834b033344897a353bf5cc0e1d226" is_pre_election: true
I20260927 21:12:32.362221 4751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e10ca764eb43d3b175e99620e4cfa2" candidate_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273df4d30eda449494b5a9ea9ff030b8" is_pre_election: true
I20260927 21:12:32.362357 4751 raft_consensus.cc:2454] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0f36da9ca24fde8d95e0192bf600f0 in term 0.
I20260927 21:12:32.362358 4618 raft_consensus.cc:2454] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0f36da9ca24fde8d95e0192bf600f0 in term 0.
I20260927 21:12:32.362596 4817 leader_election.cc:302] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0f36da9ca24fde8d95e0192bf600f0, 13e834b033344897a353bf5cc0e1d226; no voters:
I20260927 21:12:32.362735 4955 raft_consensus.cc:2781] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:32.362798 4955 raft_consensus.cc:482] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:32.362814 4955 raft_consensus.cc:3037] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:32.363551 4955 raft_consensus.cc:504] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } }
I20260927 21:12:32.363703 4955 leader_election.cc:288] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [CANDIDATE]: Term 1 election: Requested vote from peers 13e834b033344897a353bf5cc0e1d226 (127.0.98.1:36721), 273df4d30eda449494b5a9ea9ff030b8 (127.0.98.2:43169)
I20260927 21:12:32.363862 4618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e10ca764eb43d3b175e99620e4cfa2" candidate_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13e834b033344897a353bf5cc0e1d226"
I20260927 21:12:32.363893 4751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e10ca764eb43d3b175e99620e4cfa2" candidate_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273df4d30eda449494b5a9ea9ff030b8"
I20260927 21:12:32.363957 4751 raft_consensus.cc:3037] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:32.363945 4618 raft_consensus.cc:3037] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:32.364641 4751 raft_consensus.cc:2454] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0f36da9ca24fde8d95e0192bf600f0 in term 1.
I20260927 21:12:32.364705 4618 raft_consensus.cc:2454] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0f36da9ca24fde8d95e0192bf600f0 in term 1.
I20260927 21:12:32.364818 4819 leader_election.cc:302] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0f36da9ca24fde8d95e0192bf600f0, 273df4d30eda449494b5a9ea9ff030b8; no voters:
I20260927 21:12:32.364912 4955 raft_consensus.cc:2781] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:32.365077 4955 raft_consensus.cc:686] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 1 LEADER]: Becoming Leader. State: Replica: 0e0f36da9ca24fde8d95e0192bf600f0, State: Running, Role: LEADER
I20260927 21:12:32.365168 4955 consensus_queue.cc:237] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } }
I20260927 21:12:32.366154 4473 catalog_manager.cc:5495] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e0f36da9ca24fde8d95e0192bf600f0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e0f36da9ca24fde8d95e0192bf600f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42781 } health_report { overall_health: HEALTHY } } }
I20260927 21:12:32.371557 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.369181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.0.98.3:42781)
0927 21:12:32.369222 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.369224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.369232 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:32.369311 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.369311 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.369330 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.369390 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:32.369502 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.369972 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.370783 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:32.370984 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:32.371051 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:32.371248 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:32.371355 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:32.371433 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.371446 (+ 13us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.371468 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:12:32.373378 4751 raft_consensus.cc:1261] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [term 1 FOLLOWER]: Refusing update from remote peer 0e0f36da9ca24fde8d95e0192bf600f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:32.373378 4618 raft_consensus.cc:1261] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226 [term 1 FOLLOWER]: Refusing update from remote peer 0e0f36da9ca24fde8d95e0192bf600f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:32.373726 4957 consensus_queue.cc:1035] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273df4d30eda449494b5a9ea9ff030b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:32.373937 4955 consensus_queue.cc:1035] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13e834b033344897a353bf5cc0e1d226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:12:32.374890 4798 tablet.cc:2369] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:32.376289 4960 mvcc.cc:205] Tried to move back new op lower bound from 7334066390518669312 to 7334066390488584192. Current Snapshot: MvccSnapshot[applied={T|T < 7334066390518669312}]
I20260927 21:12:32.378947 4474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:32.379071 4474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:32.379892 4474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92983374afa64e2593023ffd0b45cc1f: Sending DeleteTablet for 3 replicas of tablet 10e10ca764eb43d3b175e99620e4cfa2
I20260927 21:12:32.380270 4864 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e10ca764eb43d3b175e99620e4cfa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37598
I20260927 21:12:32.380306 4598 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e10ca764eb43d3b175e99620e4cfa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40924
I20260927 21:12:32.380398 4731 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e10ca764eb43d3b175e99620e4cfa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60878
I20260927 21:12:32.380638 4971 tablet_replica.cc:330] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226: stopping tablet replica
I20260927 21:12:32.380779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 4536
I20260927 21:12:32.380956 4971 raft_consensus.cc:2229] T 10e10ca764eb43d3b175e99620e4cfa2 P 13e834b033344897a353bf5cc0e1d226 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:32.381448 4970 tablet_replica.cc:330] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: stopping tablet replica
I20260927 21:12:32.381552 4970 raft_consensus.cc:2229] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:32.381649 4970 raft_consensus.cc:2258] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:32.382259 4970 ts_tablet_manager.cc:1905] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:32.384268 4970 ts_tablet_manager.cc:1918] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:32.384371 4970 log.cc:1196] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/10e10ca764eb43d3b175e99620e4cfa2
I20260927 21:12:32.384642 4970 ts_tablet_manager.cc:1939] T 10e10ca764eb43d3b175e99620e4cfa2 P 0e0f36da9ca24fde8d95e0192bf600f0: Deleting consensus metadata
I20260927 21:12:32.384884 4972 tablet_replica.cc:330] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8: stopping tablet replica
I20260927 21:12:32.385017 4972 raft_consensus.cc:2229] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:32.385150 4972 raft_consensus.cc:2258] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:32.385692 4972 ts_tablet_manager.cc:1905] T 10e10ca764eb43d3b175e99620e4cfa2 P 273df4d30eda449494b5a9ea9ff030b8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:12:32.386540 4458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36721 (error 108)
I20260927 21:12:32.386775 4461 catalog_manager.cc:4841] TS 0e0f36da9ca24fde8d95e0192bf600f0 (127.0.98.3:42781): tablet 10e10ca764eb43d3b175e99620e4cfa2 (table test-table [id=ab04e69da333410da04508e885759892]) successfully deleted
I20260927 21:12:32.387051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 4669
I20260927 21:12:32.387892 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.387216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36721 (local address 127.0.0.1:40930)
0927 21:12:32.387812 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.387855 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574}
W20260927 21:12:32.389777 4458 catalog_manager.cc:4584] TS 13e834b033344897a353bf5cc0e1d226 (127.0.98.1:36721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10e10ca764eb43d3b175e99620e4cfa2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36721: connect: Connection refused (error 111)
I20260927 21:12:32.392473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 4802
I20260927 21:12:32.394220 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.392635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43169 (local address 127.0.0.1:60882)
0927 21:12:32.394135 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.394178 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1470}
I20260927 21:12:32.397301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 4443
2026-09-27T21:12:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1876 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T21:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:32Z Disabled control of system clock
I20260927 21:12:32.423486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40787
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:12:32.504551 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:32.504784 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:32.504807 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:32.506101 4977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:32.506160 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:32.506171 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:32.506182 4977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:32.506194 4977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:32.507671 4977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:32.507941 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:32.508251 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:32.513185 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:32.513397 4977 server_base.cc:1034] running on GCE node
W20260927 21:12:32.513192 4983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.513192 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:32.513756 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:32.514050 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:32.515211 4977 hybrid_clock.cc:648] HybridClock initialized: now 1790543552515194 us; error 28 us; skew 500 ppm
I20260927 21:12:32.516475 4977 webserver.cc:460] Webserver started at http://127.0.98.62:34775/ using document root <none> and password file <none>
I20260927 21:12:32.516698 4977 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:32.516749 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:32.516880 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:32.518079 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "3fc75591247849eabbdda68937b53340"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "304c30981fe9030424830e8fddd1f906"
server_key_iv: "a11fb7ec114472125b96a4688b2d3fb0"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.518419 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "3fc75591247849eabbdda68937b53340"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "304c30981fe9030424830e8fddd1f906"
server_key_iv: "a11fb7ec114472125b96a4688b2d3fb0"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.521690 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 21:12:32.522742 4991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.522934 4977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:32.523039 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "3fc75591247849eabbdda68937b53340"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "304c30981fe9030424830e8fddd1f906"
server_key_iv: "a11fb7ec114472125b96a4688b2d3fb0"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.523125 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:32.527446 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:32.527683 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:32.527768 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:32.531386 4977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40787
I20260927 21:12:32.531440 5043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40787 every 8 connection(s)
I20260927 21:12:32.531759 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:32.532409 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:32.535164 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340: Bootstrap starting.
I20260927 21:12:32.535729 5044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:32.535996 5044 log.cc:824] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:32.536650 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340: No bootstrap required, opened a new log
I20260927 21:12:32.538033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 4977
I20260927 21:12:32.538110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:32.538264 392 external_mini_cluster.cc:1428] Setting key 1a661ab235c3292e0ea924a5f7fbd32c
I20260927 21:12:32.538498 5044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc75591247849eabbdda68937b53340" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } }
I20260927 21:12:32.538681 5044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:32.538744 5044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc75591247849eabbdda68937b53340, State: Initialized, Role: FOLLOWER
I20260927 21:12:32.538888 5044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc75591247849eabbdda68937b53340" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } }
I20260927 21:12:32.538988 5044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:32.539021 5044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:32.539065 5044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:32.539794 5044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc75591247849eabbdda68937b53340" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } }
I20260927 21:12:32.539888 5044 leader_election.cc:302] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fc75591247849eabbdda68937b53340; no voters:
I20260927 21:12:32.540083 5044 leader_election.cc:288] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:32.540122 5049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:32.540266 5049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [term 1 LEADER]: Becoming Leader. State: Replica: 3fc75591247849eabbdda68937b53340, State: Running, Role: LEADER
I20260927 21:12:32.540360 5044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:32.540369 5049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc75591247849eabbdda68937b53340" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } }
I20260927 21:12:32.540817 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.538771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.0.98.62:40787)
0927 21:12:32.539002 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.539012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.539048 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:12:32.539166 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.539173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.539229 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.539374 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.539979 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.539983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.539983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:32.539988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.540033 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:32.540045 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.540101 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.540294 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.540356 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260927 21:12:32.541009 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fc75591247849eabbdda68937b53340. Latest consensus state: current_term: 1 leader_uuid: "3fc75591247849eabbdda68937b53340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc75591247849eabbdda68937b53340" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } } }
I20260927 21:12:32.541008 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fc75591247849eabbdda68937b53340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc75591247849eabbdda68937b53340" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40787 } } }
I20260927 21:12:32.541112 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:32.541119 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:32.541541 5054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:32.542109 5054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:32.543821 5054 catalog_manager.cc:1347] Generated new cluster ID: 458a5e64fbe849e48fc7e0d286a76f09
I20260927 21:12:32.543876 5054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:32.567664 5054 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:32.567845 5054 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:32.569727 5054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340: Generated new TSK 0
I20260927 21:12:32.569948 5054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:12:32.574354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:12:32.658494 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:32.658740 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:32.658773 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:32.660178 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:32.660239 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:32.660279 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:32.661991 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:32.662276 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:32.662588 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:32.667963 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.668090 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.667979 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:32.667981 5068 server_base.cc:1034] running on GCE node
I20260927 21:12:32.668432 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:32.668715 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:32.669893 5068 hybrid_clock.cc:648] HybridClock initialized: now 1790543552669873 us; error 29 us; skew 500 ppm
I20260927 21:12:32.671439 5068 webserver.cc:460] Webserver started at http://127.0.98.1:33723/ using document root <none> and password file <none>
I20260927 21:12:32.671717 5068 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:32.671780 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:32.671898 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:32.673243 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "1d17d67b15974da691a9a5d3608c4419"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "824a42ad31d9566e5e3791d6f3ed3942"
server_key_iv: "87ef20a4c12d91a92286b47231873c6c"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.673689 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1d17d67b15974da691a9a5d3608c4419"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "824a42ad31d9566e5e3791d6f3ed3942"
server_key_iv: "87ef20a4c12d91a92286b47231873c6c"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.675199 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:32.675969 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.676160 5068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:32.676260 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "1d17d67b15974da691a9a5d3608c4419"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "824a42ad31d9566e5e3791d6f3ed3942"
server_key_iv: "87ef20a4c12d91a92286b47231873c6c"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.676326 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:32.679769 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:32.680047 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:32.680176 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:32.680373 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:32.680685 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:32.680718 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.680747 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:32.680763 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.686429 5068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35373
I20260927 21:12:32.686512 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35373 every 8 connection(s)
I20260927 21:12:32.686836 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:12:32.689749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 5068
I20260927 21:12:32.689942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:32.690039 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.687551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35537 (local address 127.0.98.62:40787)
0927 21:12:32.687610 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.687612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.687971 (+ 359us) server_negotiation.cc:407] Connection header received
0927 21:12:32.688498 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.688499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.688509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.688558 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.689784 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.689786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.689786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:32.689792 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.689810 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:32.689815 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.689865 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.689950 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.689969 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:12:32.690168 392 external_mini_cluster.cc:1428] Setting key a86068871bf37c44741dbbfcd9c71368
I20260927 21:12:32.690317 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.687542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40787 (local address 127.0.98.1:35537)
0927 21:12:32.687816 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.687873 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.688382 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.688644 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.688648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.688659 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:32.689614 (+ 955us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:12:32.689615 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:32.689650 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:12:32.689682 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:32.689871 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:32.689877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:32.689881 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.689955 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.690022 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260927 21:12:32.690816 5196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40787
I20260927 21:12:32.690934 5196 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:32.691090 5196 heartbeater.cc:511] Master 127.0.98.62:40787 requested a full tablet report, sending...
I20260927 21:12:32.691141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 21:12:32.691622 5008 ts_manager.cc:194] Registered new tserver with Master: 1d17d67b15974da691a9a5d3608c4419 (127.0.98.1:35373)
W20260927 21:12:32.773708 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:32.773926 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:32.773948 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:32.775447 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:32.775506 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:32.775542 5199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:32.777019 5199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:32.777300 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:32.777601 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:12:32.782650 5199 server_base.cc:1034] running on GCE node
W20260927 21:12:32.782614 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.782614 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.782612 5207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:32.783068 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:32.783318 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:32.784497 5199 hybrid_clock.cc:648] HybridClock initialized: now 1790543552784458 us; error 32 us; skew 500 ppm
I20260927 21:12:32.785849 5199 webserver.cc:460] Webserver started at http://127.0.98.2:41041/ using document root <none> and password file <none>
I20260927 21:12:32.786079 5199 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:32.786134 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:32.786262 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:32.787312 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad2da0277c5c4d5f8b700953f85ab873"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "4e9a7d6c9c4cd4d5400ca8b9ec6da757"
server_key_iv: "51d4046e83b491c70912858538e68b3e"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.787650 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad2da0277c5c4d5f8b700953f85ab873"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "4e9a7d6c9c4cd4d5400ca8b9ec6da757"
server_key_iv: "51d4046e83b491c70912858538e68b3e"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.788929 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:12:32.789625 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.789783 5199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.789863 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "ad2da0277c5c4d5f8b700953f85ab873"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "4e9a7d6c9c4cd4d5400ca8b9ec6da757"
server_key_iv: "51d4046e83b491c70912858538e68b3e"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.789927 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:32.793296 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:32.793584 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:32.793708 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:32.793900 5199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:32.794201 5199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:32.794234 5199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.794260 5199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:32.794286 5199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.799997 5199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45951
I20260927 21:12:32.800036 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45951 every 8 connection(s)
I20260927 21:12:32.800357 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:12:32.803431 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.801127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59163 (local address 127.0.98.62:40787)
0927 21:12:32.801204 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.801206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.801402 (+ 196us) server_negotiation.cc:407] Connection header received
0927 21:12:32.801915 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.801916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.801922 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.801988 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.803142 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.803144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.803144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:32.803149 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.803166 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:32.803182 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.803243 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.803360 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.803373 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:12:32.803635 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.801072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40787 (local address 127.0.98.2:59163)
0927 21:12:32.801260 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.801331 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.801803 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.802050 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.802053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.802062 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:32.802973 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:12:32.802974 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:32.803004 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:12:32.803034 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:32.803276 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:32.803283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:32.803286 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.803368 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.803425 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260927 21:12:32.804071 5327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40787
I20260927 21:12:32.804165 5327 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:32.804291 5327 heartbeater.cc:511] Master 127.0.98.62:40787 requested a full tablet report, sending...
I20260927 21:12:32.804642 5008 ts_manager.cc:194] Registered new tserver with Master: ad2da0277c5c4d5f8b700953f85ab873 (127.0.98.2:45951)
I20260927 21:12:32.806052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 5199
I20260927 21:12:32.806124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:32.806267 392 external_mini_cluster.cc:1428] Setting key 64b05746b666feff6a268293c6478d7d
I20260927 21:12:32.807221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--builtin_ntp_servers=127.0.98.20:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:12:32.887910 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:32.888140 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:32.888163 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:32.889411 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:32.889456 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:32.889482 5330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:32.891134 5330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:32.891361 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:32.891682 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:32.896723 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.896723 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:32.896752 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:32.896939 5330 server_base.cc:1034] running on GCE node
I20260927 21:12:32.897153 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:32.897413 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:32.898555 5330 hybrid_clock.cc:648] HybridClock initialized: now 1790543552898544 us; error 27 us; skew 500 ppm
I20260927 21:12:32.899888 5330 webserver.cc:460] Webserver started at http://127.0.98.3:46599/ using document root <none> and password file <none>
I20260927 21:12:32.900117 5330 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:32.900166 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:32.900309 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:32.901453 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "db3ea577249e4d979bb651f95146cbfc"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "427f36f66d8f785f437e541b661a675d"
server_key_iv: "552bd4d40d5df2105a857729ce364467"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.901839 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "db3ea577249e4d979bb651f95146cbfc"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "427f36f66d8f785f437e541b661a675d"
server_key_iv: "552bd4d40d5df2105a857729ce364467"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.903154 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:32.903945 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.904093 5330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:32.904179 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "db3ea577249e4d979bb651f95146cbfc"
format_stamp: "Formatted at 2026-09-27 21:12:32 on dist-test-slave-r9x1"
server_key: "427f36f66d8f785f437e541b661a675d"
server_key_iv: "552bd4d40d5df2105a857729ce364467"
server_key_version: "encryptionkey@0"
I20260927 21:12:32.904260 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:32.907807 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:32.908129 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:32.908257 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:32.908484 5330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:32.908800 5330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:32.908833 5330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.908862 5330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:32.908885 5330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:32.914690 5330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36897
I20260927 21:12:32.914785 5457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36897 every 8 connection(s)
I20260927 21:12:32.915091 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:32.918131 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.915833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56373 (local address 127.0.98.62:40787)
0927 21:12:32.915918 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.915920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.916108 (+ 188us) server_negotiation.cc:407] Connection header received
0927 21:12:32.916608 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.916610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.916617 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.916664 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.917878 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.917880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.917881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:32.917885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.917902 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:32.917906 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.917958 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.918040 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.918056 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:12:32.918334 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.915792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40787 (local address 127.0.98.3:56373)
0927 21:12:32.915982 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.916020 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.916502 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.916757 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.916762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.916772 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:32.917716 (+ 944us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:12:32.917717 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:32.917748 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:12:32.917778 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:32.917964 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:32.917969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:32.917972 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.918045 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.918111 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260927 21:12:32.918870 5458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40787
I20260927 21:12:32.918962 5458 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:32.919098 5458 heartbeater.cc:511] Master 127.0.98.62:40787 requested a full tablet report, sending...
I20260927 21:12:32.919480 5008 ts_manager.cc:194] Registered new tserver with Master: db3ea577249e4d979bb651f95146cbfc (127.0.98.3:36897)
I20260927 21:12:32.921988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 5330
I20260927 21:12:32.922087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:32.922254 392 external_mini_cluster.cc:1428] Setting key 68551cdc47a5527569547e314c304d77
I20260927 21:12:32.923161 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:12:32.925473 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.924432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.98.62:40787)
0927 21:12:32.924493 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.924495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.924567 (+ 72us) server_negotiation.cc:407] Connection header received
0927 21:12:32.924643 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.924644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.924649 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.924690 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.925238 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.925239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.925240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:32.925243 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.925284 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:32.925289 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.925331 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.925386 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.925401 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:12:32.927052 5008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:32.928634 5008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:32.932133 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.930690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35373 (local address 127.0.0.1:57700)
0927 21:12:32.930882 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.930912 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.930962 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.931155 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.931160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.931166 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:32.931735 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:12:32.931735 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:32.931753 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:12:32.931778 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:32.931980 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:32.931985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:32.931988 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.932045 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.932058 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260927 21:12:32.932164 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.930859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.0.98.1:35373)
0927 21:12:32.931036 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.931042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.931054 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:12:32.931078 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.931081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.931090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.931157 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.931860 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.931869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.931869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:32.931873 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.931909 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:32.931935 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.931988 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.932053 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.932066 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260927 21:12:32.932415 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.931032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.0.98.2:45951)
0927 21:12:32.931184 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.931188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.931198 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:32.931223 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.931225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.931234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.931497 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.931952 (+ 455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.931960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.931960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:32.931965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.931998 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:32.932024 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.932295 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.932320 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.932335 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260927 21:12:32.932224 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.930690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45951 (local address 127.0.0.1:58288)
0927 21:12:32.930882 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.930915 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.930963 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.931335 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.931336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.931339 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:32.931863 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:12:32.931863 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:32.931869 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:12:32.931878 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:32.932111 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:32.932112 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:32.932112 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.932142 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.932154 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260927 21:12:32.933105 5130 tablet_service.cc:1467] Processing CreateTablet for tablet 352b2b5e6b034ca6beb2503938c5d29a (DEFAULT_TABLE table=test-table [id=55b00e61d37c42ebba292b12abd0de7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:32.933236 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 352b2b5e6b034ca6beb2503938c5d29a (DEFAULT_TABLE table=test-table [id=55b00e61d37c42ebba292b12abd0de7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:32.933467 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352b2b5e6b034ca6beb2503938c5d29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:32.933580 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352b2b5e6b034ca6beb2503938c5d29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:32.936071 5472 tablet_bootstrap.cc:492] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419: Bootstrap starting.
I20260927 21:12:32.936453 5473 tablet_bootstrap.cc:492] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: Bootstrap starting.
I20260927 21:12:32.936753 5472 tablet_bootstrap.cc:654] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:32.936822 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.930696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36897 (local address 127.0.0.1:54894)
0927 21:12:32.931035 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:32.931042 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:32.931072 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:32.932321 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:32.932322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:32.932325 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:32.936336 (+ 4011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:12:32.936336 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:32.936347 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:12:32.936368 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:32.936670 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:32.936671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:32.936672 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:12:32.936721 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:12:32.936738 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":103,"threads_started":1}
I20260927 21:12:32.937032 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:32.931234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.0.98.3:36897)
0927 21:12:32.931938 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:32.931942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:32.931951 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:32.931974 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:32.931977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:32.931984 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:32.932059 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:32.936464 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:32.936476 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:32.936476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:32.936483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:32.936523 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:32.936563 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:32.936628 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:32.936933 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 21:12:32.936952 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":75,"threads_started":1}
I20260927 21:12:32.937080 5473 tablet_bootstrap.cc:654] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:32.937193 5472 log.cc:824] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:32.937422 5473 log.cc:824] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:32.937758 5392 tablet_service.cc:1467] Processing CreateTablet for tablet 352b2b5e6b034ca6beb2503938c5d29a (DEFAULT_TABLE table=test-table [id=55b00e61d37c42ebba292b12abd0de7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:32.937968 5472 tablet_bootstrap.cc:492] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419: No bootstrap required, opened a new log
I20260927 21:12:32.938086 5472 ts_tablet_manager.cc:1397] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:32.938113 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352b2b5e6b034ca6beb2503938c5d29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:32.938145 5473 tablet_bootstrap.cc:492] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: No bootstrap required, opened a new log
I20260927 21:12:32.938238 5473 ts_tablet_manager.cc:1397] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:32.940640 5476 tablet_bootstrap.cc:492] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: Bootstrap starting.
I20260927 21:12:32.940701 5472 raft_consensus.cc:348] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } }
I20260927 21:12:32.940769 5473 raft_consensus.cc:348] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } }
I20260927 21:12:32.940892 5473 raft_consensus.cc:374] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:32.940891 5472 raft_consensus.cc:374] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:32.940932 5472 raft_consensus.cc:729] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d17d67b15974da691a9a5d3608c4419, State: Initialized, Role: FOLLOWER
I20260927 21:12:32.940960 5473 raft_consensus.cc:729] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2da0277c5c4d5f8b700953f85ab873, State: Initialized, Role: FOLLOWER
I20260927 21:12:32.941095 5473 consensus_queue.cc:260] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } }
I20260927 21:12:32.941095 5472 consensus_queue.cc:260] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } }
I20260927 21:12:32.941394 5473 ts_tablet_manager.cc:1428] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:12:32.941394 5472 ts_tablet_manager.cc:1428] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:12:32.941411 5476 tablet_bootstrap.cc:654] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:32.941541 5327 heartbeater.cc:503] Master 127.0.98.62:40787 was elected leader, sending a full tablet report...
I20260927 21:12:32.941757 5476 log.cc:824] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:32.941856 5196 heartbeater.cc:503] Master 127.0.98.62:40787 was elected leader, sending a full tablet report...
I20260927 21:12:32.942723 5476 tablet_bootstrap.cc:492] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: No bootstrap required, opened a new log
I20260927 21:12:32.942862 5476 ts_tablet_manager.cc:1397] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:12:32.944818 5476 raft_consensus.cc:348] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } }
I20260927 21:12:32.944971 5476 raft_consensus.cc:374] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:32.944993 5476 raft_consensus.cc:729] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3ea577249e4d979bb651f95146cbfc, State: Initialized, Role: FOLLOWER
I20260927 21:12:32.945101 5476 consensus_queue.cc:260] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } }
I20260927 21:12:32.945399 5476 ts_tablet_manager.cc:1428] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 21:12:32.945451 5458 heartbeater.cc:503] Master 127.0.98.62:40787 was elected leader, sending a full tablet report...
W20260927 21:12:33.050909 5328 tablet.cc:2369] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:12:33.165683 5459 tablet.cc:2369] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:12:33.187578 5197 tablet.cc:2369] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:33.229564 5480 raft_consensus.cc:482] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:33.229704 5480 raft_consensus.cc:504] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } }
I20260927 21:12:33.230083 5480 leader_election.cc:288] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d17d67b15974da691a9a5d3608c4419 (127.0.98.1:35373), ad2da0277c5c4d5f8b700953f85ab873 (127.0.98.2:45951)
I20260927 21:12:33.231442 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.230306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45951 (local address 127.0.98.3:44157)
0927 21:12:33.230424 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:33.230437 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:33.230476 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:33.230640 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:33.230640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:33.230643 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:33.231054 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:12:33.231055 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:33.231062 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:12:33.231073 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:33.231291 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:33.231291 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:33.231293 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:12:33.231354 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:12:33.231369 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 21:12:33.231534 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.230439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44157 (local address 127.0.98.2:45951)
0927 21:12:33.230490 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:33.230491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:33.230502 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:12:33.230532 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:33.230533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:33.230540 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:33.230619 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:33.231167 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:33.231169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:33.231169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:33.231173 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:33.231191 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:33.231196 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:33.231244 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:33.231452 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 21:12:33.231466 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:12:33.231561 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.230306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35373 (local address 127.0.98.3:39265)
0927 21:12:33.230600 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:33.230609 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:33.230657 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:33.230781 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:33.230782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:33.230796 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:33.231181 (+ 385us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:12:33.231181 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:33.231200 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:12:33.231284 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:33.231432 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:33.231432 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:33.231433 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:12:33.231477 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 21:12:33.231487 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":129,"thread_start_us":98,"threads_started":1}
I20260927 21:12:33.231567 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.230493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39265 (local address 127.0.98.1:35373)
0927 21:12:33.230554 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:33.230555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:33.230691 (+ 136us) server_negotiation.cc:407] Connection header received
0927 21:12:33.230714 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:33.230715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:33.230719 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:33.230756 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:33.231371 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:33.231372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:33.231373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:33.231376 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:33.231390 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:33.231394 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:33.231431 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:33.231497 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 21:12:33.231509 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:12:33.231851 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352b2b5e6b034ca6beb2503938c5d29a" candidate_uuid: "db3ea577249e4d979bb651f95146cbfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d17d67b15974da691a9a5d3608c4419" is_pre_election: true
I20260927 21:12:33.231855 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352b2b5e6b034ca6beb2503938c5d29a" candidate_uuid: "db3ea577249e4d979bb651f95146cbfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2da0277c5c4d5f8b700953f85ab873" is_pre_election: true
I20260927 21:12:33.232009 5150 raft_consensus.cc:2454] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db3ea577249e4d979bb651f95146cbfc in term 0.
I20260927 21:12:33.232009 5281 raft_consensus.cc:2454] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db3ea577249e4d979bb651f95146cbfc in term 0.
I20260927 21:12:33.232244 5347 leader_election.cc:302] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d17d67b15974da691a9a5d3608c4419, db3ea577249e4d979bb651f95146cbfc; no voters:
I20260927 21:12:33.232383 5480 raft_consensus.cc:2781] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:33.232462 5480 raft_consensus.cc:482] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:33.232481 5480 raft_consensus.cc:3037] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:33.233186 5480 raft_consensus.cc:504] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } }
I20260927 21:12:33.233379 5480 leader_election.cc:288] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [CANDIDATE]: Term 1 election: Requested vote from peers 1d17d67b15974da691a9a5d3608c4419 (127.0.98.1:35373), ad2da0277c5c4d5f8b700953f85ab873 (127.0.98.2:45951)
I20260927 21:12:33.233563 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352b2b5e6b034ca6beb2503938c5d29a" candidate_uuid: "db3ea577249e4d979bb651f95146cbfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d17d67b15974da691a9a5d3608c4419"
I20260927 21:12:33.233610 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352b2b5e6b034ca6beb2503938c5d29a" candidate_uuid: "db3ea577249e4d979bb651f95146cbfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2da0277c5c4d5f8b700953f85ab873"
I20260927 21:12:33.233664 5150 raft_consensus.cc:3037] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:33.233676 5281 raft_consensus.cc:3037] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:33.234560 5150 raft_consensus.cc:2454] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db3ea577249e4d979bb651f95146cbfc in term 1.
I20260927 21:12:33.234560 5281 raft_consensus.cc:2454] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db3ea577249e4d979bb651f95146cbfc in term 1.
I20260927 21:12:33.234772 5347 leader_election.cc:302] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d17d67b15974da691a9a5d3608c4419, db3ea577249e4d979bb651f95146cbfc; no voters:
I20260927 21:12:33.234851 5480 raft_consensus.cc:2781] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:33.234984 5480 raft_consensus.cc:686] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 1 LEADER]: Becoming Leader. State: Replica: db3ea577249e4d979bb651f95146cbfc, State: Running, Role: LEADER
I20260927 21:12:33.235080 5480 consensus_queue.cc:237] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } }
I20260927 21:12:33.235865 5008 catalog_manager.cc:5495] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc reported cstate change: term changed from 0 to 1, leader changed from <none> to db3ea577249e4d979bb651f95146cbfc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "db3ea577249e4d979bb651f95146cbfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db3ea577249e4d979bb651f95146cbfc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } health_report { overall_health: UNKNOWN } } }
I20260927 21:12:33.287770 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.286566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.0.98.3:36897)
0927 21:12:33.286658 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:33.286660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:33.286669 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:33.286745 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:33.286746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:33.286855 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:33.286934 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:33.287509 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:33.287511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:33.287511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:33.287516 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:33.287531 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:33.287535 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:33.287584 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:33.287676 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:12:33.287689 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:12:33.289592 5281 raft_consensus.cc:1261] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [term 1 FOLLOWER]: Refusing update from remote peer db3ea577249e4d979bb651f95146cbfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:33.289654 5150 raft_consensus.cc:1261] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [term 1 FOLLOWER]: Refusing update from remote peer db3ea577249e4d979bb651f95146cbfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:33.289979 5482 consensus_queue.cc:1035] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d17d67b15974da691a9a5d3608c4419" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:33.290108 5480 consensus_queue.cc:1035] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2da0277c5c4d5f8b700953f85ab873" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:33.292444 5485 mvcc.cc:205] Tried to move back new op lower bound from 7334066394271240192 to 7334066394051502080. Current Snapshot: MvccSnapshot[applied={T|T < 7334066394271240192}]
I20260927 21:12:33.293396 5486 mvcc.cc:205] Tried to move back new op lower bound from 7334066394271240192 to 7334066394051502080. Current Snapshot: MvccSnapshot[applied={T|T < 7334066394271240192 or (T in {7334066394271240192})}]
I20260927 21:12:33.294732 5006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:33.294854 5006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:33.294924 5488 mvcc.cc:205] Tried to move back new op lower bound from 7334066394271240192 to 7334066394051502080. Current Snapshot: MvccSnapshot[applied={T|T < 7334066394271240192 or (T in {7334066394271240192})}]
I20260927 21:12:33.295709 5006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fc75591247849eabbdda68937b53340: Sending DeleteTablet for 3 replicas of tablet 352b2b5e6b034ca6beb2503938c5d29a
I20260927 21:12:33.295969 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet 352b2b5e6b034ca6beb2503938c5d29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:33 UTC) from {username='slave'} at 127.0.0.1:54894
I20260927 21:12:33.295958 5261 tablet_service.cc:1514] Processing DeleteTablet for tablet 352b2b5e6b034ca6beb2503938c5d29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:33 UTC) from {username='slave'} at 127.0.0.1:58288
I20260927 21:12:33.296149 5260 tablet_service.cc:1514] Processing DeleteTablet for tablet 352b2b5e6b034ca6beb2503938c5d29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:33 UTC) from {username='slave'} at 127.0.0.1:58288
I20260927 21:12:33.296211 5495 tablet_replica.cc:330] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: stopping tablet replica
I20260927 21:12:33.296320 5495 raft_consensus.cc:2229] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:33.296416 5495 raft_consensus.cc:2258] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:33.296480 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet 352b2b5e6b034ca6beb2503938c5d29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:33 UTC) from {username='slave'} at 127.0.0.1:57700
I20260927 21:12:33.296566 5494 tablet_replica.cc:330] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: stopping tablet replica
I20260927 21:12:33.296656 5494 raft_consensus.cc:2229] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:33.296700 5496 tablet_replica.cc:330] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419: stopping tablet replica
I20260927 21:12:33.296749 5494 raft_consensus.cc:2258] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:33.296792 5496 raft_consensus.cc:2229] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:33.296808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 5068
I20260927 21:12:33.296896 5496 raft_consensus.cc:2258] T 352b2b5e6b034ca6beb2503938c5d29a P 1d17d67b15974da691a9a5d3608c4419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:33.297257 5495 ts_tablet_manager.cc:1905] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:33.298979 5495 ts_tablet_manager.cc:1918] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:33.299050 5495 log.cc:1196] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/352b2b5e6b034ca6beb2503938c5d29a
I20260927 21:12:33.299316 5495 ts_tablet_manager.cc:1939] T 352b2b5e6b034ca6beb2503938c5d29a P db3ea577249e4d979bb651f95146cbfc: Deleting consensus metadata
I20260927 21:12:33.299744 4992 catalog_manager.cc:4841] TS db3ea577249e4d979bb651f95146cbfc (127.0.98.3:36897): tablet 352b2b5e6b034ca6beb2503938c5d29a (table test-table [id=55b00e61d37c42ebba292b12abd0de7f]) successfully deleted
I20260927 21:12:33.300158 5494 ts_tablet_manager.cc:1905] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:33.302021 5494 ts_tablet_manager.cc:1918] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:33.302130 5494 log.cc:1196] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/352b2b5e6b034ca6beb2503938c5d29a
I20260927 21:12:33.302420 5494 ts_tablet_manager.cc:1939] T 352b2b5e6b034ca6beb2503938c5d29a P ad2da0277c5c4d5f8b700953f85ab873: Deleting consensus metadata
W20260927 21:12:33.302472 4994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35373 (error 108)
I20260927 21:12:33.302717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 5199
I20260927 21:12:33.303575 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.302892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35373 (local address 127.0.0.1:57714)
0927 21:12:33.303496 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:33.303537 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583}
W20260927 21:12:33.303699 4994 catalog_manager.cc:4584] TS 1d17d67b15974da691a9a5d3608c4419 (127.0.98.1:35373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 352b2b5e6b034ca6beb2503938c5d29a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35373: connect: Connection refused (error 111)
I20260927 21:12:33.307952 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.307805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45951 (local address 127.0.0.1:58290)
0927 21:12:33.307866 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:33.307906 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 21:12:33.307960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 5330
I20260927 21:12:33.312067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 4977
2026-09-27T21:12:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (914 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 21:12:33.331089 392 mini_oidc.cc:152] Starting JWKS server
I20260927 21:12:33.332170 392 webserver.cc:460] Webserver started at https://127.0.0.1:41935/ using document root <none> and password file <none>
I20260927 21:12:33.332258 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:12:33.332392 392 webserver.cc:460] Webserver started at http://127.0.0.1:42545/ using document root <none> and password file <none>
2026-09-27T21:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:33Z Disabled control of system clock
I20260927 21:12:33.340018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34361
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34361
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41935/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/testchainca.cert with env {}
W20260927 21:12:33.424198 5503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:33.424435 5503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:33.424468 5503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:33.425797 5503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:33.425849 5503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:33.425868 5503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:33.425884 5503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:33.425899 5503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:12:33.425920 5503 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:12:33.425940 5503 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41935/jwks.json
I20260927 21:12:33.427630 5503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41935/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:33.427901 5503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:33.428117 5503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:33.433008 5511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:33.433023 5509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:33.433308 5503 server_base.cc:1034] running on GCE node
W20260927 21:12:33.433018 5508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:33.433578 5503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:33.433888 5503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:33.435074 5503 hybrid_clock.cc:648] HybridClock initialized: now 1790543553435053 us; error 39 us; skew 500 ppm
I20260927 21:12:33.436344 5503 webserver.cc:460] Webserver started at http://127.0.98.62:38103/ using document root <none> and password file <none>
I20260927 21:12:33.436576 5503 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:33.436627 5503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:33.436751 5503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:33.437824 5503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "f8b88d32d4ed437c8a73d95cfc91bec3"
format_stamp: "Formatted at 2026-09-27 21:12:33 on dist-test-slave-r9x1"
server_key: "56e7eaad4036751395dc5f1f940998b4"
server_key_iv: "3c55f7b7f7097f1d48a98341f4f382b1"
server_key_version: "encryptionkey@0"
I20260927 21:12:33.438175 5503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8b88d32d4ed437c8a73d95cfc91bec3"
format_stamp: "Formatted at 2026-09-27 21:12:33 on dist-test-slave-r9x1"
server_key: "56e7eaad4036751395dc5f1f940998b4"
server_key_iv: "3c55f7b7f7097f1d48a98341f4f382b1"
server_key_version: "encryptionkey@0"
I20260927 21:12:33.439435 5503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:33.440156 5517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:33.440335 5503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:33.440419 5503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "f8b88d32d4ed437c8a73d95cfc91bec3"
format_stamp: "Formatted at 2026-09-27 21:12:33 on dist-test-slave-r9x1"
server_key: "56e7eaad4036751395dc5f1f940998b4"
server_key_iv: "3c55f7b7f7097f1d48a98341f4f382b1"
server_key_version: "encryptionkey@0"
I20260927 21:12:33.440483 5503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:33.447288 5503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:33.447583 5503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:33.447706 5503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:33.451689 5503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34361
I20260927 21:12:33.451726 5569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34361 every 8 connection(s)
I20260927 21:12:33.452059 5503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:33.453020 5570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:33.454711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 5503
I20260927 21:12:33.454824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:33.455009 392 external_mini_cluster.cc:1428] Setting key 7ccdc0876a1c5f39bff67535be23b29e
I20260927 21:12:33.455693 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3: Bootstrap starting.
I20260927 21:12:33.456408 5570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:33.456728 5570 log.cc:824] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:33.457414 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3: No bootstrap required, opened a new log
I20260927 21:12:33.459164 5570 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b88d32d4ed437c8a73d95cfc91bec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } }
I20260927 21:12:33.459311 5570 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:33.459342 5570 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b88d32d4ed437c8a73d95cfc91bec3, State: Initialized, Role: FOLLOWER
I20260927 21:12:33.459476 5570 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b88d32d4ed437c8a73d95cfc91bec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } }
I20260927 21:12:33.459553 5570 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:33.459581 5570 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:33.459656 5570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:33.460225 5570 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b88d32d4ed437c8a73d95cfc91bec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } }
I20260927 21:12:33.460330 5570 leader_election.cc:302] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8b88d32d4ed437c8a73d95cfc91bec3; no voters:
I20260927 21:12:33.460453 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.455523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.0.98.62:34361)
0927 21:12:33.455719 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:33.455728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:33.455759 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:12:33.456295 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:33.456305 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:33.456386 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:33.456558 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:33.456742 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.457756 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.458518 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.459070 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.459228 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:33.459855 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:33.459859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:33.459860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:33.460034 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:33.460067 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:33.460078 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:33.460130 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:33.460206 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 21:12:33.460287 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260927 21:12:33.460490 5570 leader_election.cc:288] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:33.460528 5575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:33.460758 5575 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [term 1 LEADER]: Becoming Leader. State: Replica: f8b88d32d4ed437c8a73d95cfc91bec3, State: Running, Role: LEADER
I20260927 21:12:33.460860 5570 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:33.461054 5575 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b88d32d4ed437c8a73d95cfc91bec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } }
I20260927 21:12:33.461505 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:12:33.461644 5577 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8b88d32d4ed437c8a73d95cfc91bec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b88d32d4ed437c8a73d95cfc91bec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } }
I20260927 21:12:33.461683 5576 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b88d32d4ed437c8a73d95cfc91bec3. Latest consensus state: current_term: 1 leader_uuid: "f8b88d32d4ed437c8a73d95cfc91bec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b88d32d4ed437c8a73d95cfc91bec3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34361 } } }
I20260927 21:12:33.461735 5577 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:33.461748 5576 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:33.461930 5584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:33.462510 5584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:33.464222 5584 catalog_manager.cc:1347] Generated new cluster ID: e8846f27d3cd4d6a86771c64afcd8469
I20260927 21:12:33.464286 5584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:33.478420 5584 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:33.479190 5584 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:33.483567 5584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b88d32d4ed437c8a73d95cfc91bec3: Generated new TSK 0
I20260927 21:12:33.483780 5584 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:12:33.507189 5600 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 21:12:33.507288 5600 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260927 21:12:33.507921 5599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:33.491860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34361 (local address 127.0.0.1:48796)
0927 21:12:33.492032 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:33.492041 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:33.492102 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:33.492281 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:33.492281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:33.492283 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:12:33.492349 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:33.492351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.493298 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.493299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:33.493914 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:33.493916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.494376 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.494379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:33.494422 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:33.494443 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:12:33.507457 (+ 13014us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:12:33.507581 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34361: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260927 21:12:33.508378 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.491955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.0.98.62:34361)
0927 21:12:33.492001 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:33.492003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:33.492100 (+ 97us) server_negotiation.cc:407] Connection header received
0927 21:12:33.492178 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:33.492179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:33.492190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:33.492274 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:12:33.492406 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.493196 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.493993 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.494310 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.494377 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:33.494547 (+ 170us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:12:33.507292 (+ 12745us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:12:33.508175 (+ 883us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34361: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 21:12:33.508512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 5503
2026-09-27T21:12:33Z chronyd exiting
I20260927 21:12:33.543973 392 test_util.cc:175] -----------------------------------------------
I20260927 21:12:33.544061 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543529727471-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 21:12:33.544799 392 mini_oidc.cc:152] Starting JWKS server
I20260927 21:12:33.545517 392 webserver.cc:460] Webserver started at https://127.0.0.1:36843/ using document root <none> and password file <none>
I20260927 21:12:33.545567 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:12:33.545703 392 webserver.cc:460] Webserver started at http://127.0.0.1:40965/ using document root <none> and password file <none>
2026-09-27T21:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:33Z Disabled control of system clock
I20260927 21:12:33.553162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45625
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45625
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36843/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/testchainca.cert with env {}
W20260927 21:12:33.633126 5606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:33.633419 5606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:33.633446 5606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:33.634969 5606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:33.635047 5606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:33.635061 5606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:33.635073 5606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:33.635085 5606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:12:33.635097 5606 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:12:33.635106 5606 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36843/jwks.json
I20260927 21:12:33.636843 5606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36843/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:33.637090 5606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:33.637323 5606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:33.642182 5614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:33.642171 5611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:33.642203 5606 server_base.cc:1034] running on GCE node
W20260927 21:12:33.642167 5612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:33.642673 5606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:33.642987 5606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:33.644150 5606 hybrid_clock.cc:648] HybridClock initialized: now 1790543553644126 us; error 37 us; skew 500 ppm
I20260927 21:12:33.645434 5606 webserver.cc:460] Webserver started at http://127.0.98.62:35635/ using document root <none> and password file <none>
I20260927 21:12:33.645660 5606 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:33.645711 5606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:33.645838 5606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:33.646922 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "98f4fbed39334de8a64049ea33c0727b"
format_stamp: "Formatted at 2026-09-27 21:12:33 on dist-test-slave-r9x1"
server_key: "4ba3c03399268e4d757536993bc07476"
server_key_iv: "af8419277e68439c76789c83d179883a"
server_key_version: "encryptionkey@0"
I20260927 21:12:33.647276 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "98f4fbed39334de8a64049ea33c0727b"
format_stamp: "Formatted at 2026-09-27 21:12:33 on dist-test-slave-r9x1"
server_key: "4ba3c03399268e4d757536993bc07476"
server_key_iv: "af8419277e68439c76789c83d179883a"
server_key_version: "encryptionkey@0"
I20260927 21:12:33.648554 5606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:33.649313 5620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:33.649472 5606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:33.649561 5606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "98f4fbed39334de8a64049ea33c0727b"
format_stamp: "Formatted at 2026-09-27 21:12:33 on dist-test-slave-r9x1"
server_key: "4ba3c03399268e4d757536993bc07476"
server_key_iv: "af8419277e68439c76789c83d179883a"
server_key_version: "encryptionkey@0"
I20260927 21:12:33.649636 5606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:33.665676 5606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:33.666000 5606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:33.666129 5606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:33.670511 5606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45625
I20260927 21:12:33.670608 5672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45625 every 8 connection(s)
I20260927 21:12:33.670924 5606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:33.671903 5673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:33.674434 5673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b: Bootstrap starting.
I20260927 21:12:33.675104 5673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:33.675390 5673 log.cc:824] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:33.676281 5673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b: No bootstrap required, opened a new log
I20260927 21:12:33.677467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 5606
I20260927 21:12:33.677552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:33.677716 392 external_mini_cluster.cc:1428] Setting key 6189ea19b30ca4675f5f1cb311ea5e5c
I20260927 21:12:33.678058 5673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f4fbed39334de8a64049ea33c0727b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } }
I20260927 21:12:33.678256 5673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:33.678329 5673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f4fbed39334de8a64049ea33c0727b, State: Initialized, Role: FOLLOWER
I20260927 21:12:33.678478 5673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f4fbed39334de8a64049ea33c0727b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } }
I20260927 21:12:33.678558 5673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:33.678593 5673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:33.678635 5673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:33.679387 5673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f4fbed39334de8a64049ea33c0727b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } }
I20260927 21:12:33.679508 5673 leader_election.cc:302] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98f4fbed39334de8a64049ea33c0727b; no voters:
I20260927 21:12:33.679656 5673 leader_election.cc:288] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:33.679687 5678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:33.679824 5678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [term 1 LEADER]: Becoming Leader. State: Replica: 98f4fbed39334de8a64049ea33c0727b, State: Running, Role: LEADER
I20260927 21:12:33.679924 5673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:33.679951 5678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f4fbed39334de8a64049ea33c0727b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } }
I20260927 21:12:33.680334 5679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98f4fbed39334de8a64049ea33c0727b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f4fbed39334de8a64049ea33c0727b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } } }
I20260927 21:12:33.680405 5679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:33.680591 5680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98f4fbed39334de8a64049ea33c0727b. Latest consensus state: current_term: 1 leader_uuid: "98f4fbed39334de8a64049ea33c0727b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f4fbed39334de8a64049ea33c0727b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } } }
I20260927 21:12:33.680696 5680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:33.680944 5684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:33.681689 5684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:33.683178 5684 catalog_manager.cc:1347] Generated new cluster ID: b8e5661498784b8f9fac8ad079c879e9
I20260927 21:12:33.683241 5684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:33.684212 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.678225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.0.98.62:45625)
0927 21:12:33.678448 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:33.678457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:33.678487 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:12:33.678603 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:33.678610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:33.678684 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:33.678828 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:33.679074 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.680747 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.681951 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.682574 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.682697 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:33.683460 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:33.683464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:33.683464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:33.683683 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:33.683726 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:33.683743 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:33.683815 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:33.683878 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 21:12:33.683961 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260927 21:12:33.685139 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:12:33.689749 5684 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:33.690274 5684 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:33.703647 5684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98f4fbed39334de8a64049ea33c0727b: Generated new TSK 0
I20260927 21:12:33.703859 5684 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:12:33.730476 5703 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 21:12:33.730595 5703 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:12:33.730865 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.709011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.0.98.62:45625)
0927 21:12:33.709057 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:33.709059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:33.709120 (+ 61us) server_negotiation.cc:407] Connection header received
0927 21:12:33.709211 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:33.709212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:33.709224 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:33.709318 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:12:33.709455 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.710007 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.710949 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.711172 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.711231 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:33.711414 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:12:33.730631 (+ 19217us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:12:33.730799 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41404: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 21:12:33.730919 5677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41404: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 21:12:33.731191 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:33.708917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45625 (local address 127.0.0.1:41404)
0927 21:12:33.709043 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:33.709055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:33.709130 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:33.709327 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:33.709328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:33.709329 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:12:33.709393 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:33.709395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.710095 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.710096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:33.710827 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:33.710831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.711242 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.711247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:33.711309 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:33.711316 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:12:33.730926 (+ 19610us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:12:33.731065 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45625: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260927 21:12:33.731525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 5606
2026-09-27T21:12:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (223 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 21:12:33.768360 392 mini_oidc.cc:152] Starting JWKS server
I20260927 21:12:33.769042 392 webserver.cc:460] Webserver started at https://127.0.0.1:34339/ using document root <none> and password file <none>
I20260927 21:12:33.769083 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:12:33.769196 392 webserver.cc:460] Webserver started at http://127.0.0.1:39017/ using document root <none> and password file <none>
2026-09-27T21:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:33Z Disabled control of system clock
I20260927 21:12:33.776712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39973
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39973
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34339/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:12:33.857767 5709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:33.858137 5709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:33.858174 5709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:33.859570 5709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:33.859639 5709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:33.859659 5709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:33.859674 5709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:33.859691 5709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:12:33.859719 5709 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:12:33.859740 5709 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34339/jwks.json
I20260927 21:12:33.861397 5709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34339/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:33.861676 5709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:33.861982 5709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:33.866739 5717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:33.866746 5714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:33.866743 5715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:33.867022 5709 server_base.cc:1034] running on GCE node
I20260927 21:12:33.867261 5709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:33.867549 5709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:33.868705 5709 hybrid_clock.cc:648] HybridClock initialized: now 1790543553868688 us; error 34 us; skew 500 ppm
I20260927 21:12:33.870122 5709 webserver.cc:460] Webserver started at http://127.0.98.62:45087/ using document root <none> and password file <none>
I20260927 21:12:33.870398 5709 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:33.870452 5709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:33.870666 5709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:33.871811 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "b15ca94a24ec4fa8901f0b68a901ed88"
format_stamp: "Formatted at 2026-09-27 21:12:33 on dist-test-slave-r9x1"
server_key: "7bb4e52f008797c774aff13125b042c5"
server_key_iv: "172bb30c6f0adc826a5daf7660b096cb"
server_key_version: "encryptionkey@0"
I20260927 21:12:33.872167 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "b15ca94a24ec4fa8901f0b68a901ed88"
format_stamp: "Formatted at 2026-09-27 21:12:33 on dist-test-slave-r9x1"
server_key: "7bb4e52f008797c774aff13125b042c5"
server_key_iv: "172bb30c6f0adc826a5daf7660b096cb"
server_key_version: "encryptionkey@0"
I20260927 21:12:33.873648 5709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:33.874704 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:33.874902 5709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:33.874987 5709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "b15ca94a24ec4fa8901f0b68a901ed88"
format_stamp: "Formatted at 2026-09-27 21:12:33 on dist-test-slave-r9x1"
server_key: "7bb4e52f008797c774aff13125b042c5"
server_key_iv: "172bb30c6f0adc826a5daf7660b096cb"
server_key_version: "encryptionkey@0"
I20260927 21:12:33.875062 5709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:33.887996 5709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:33.888384 5709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:33.888538 5709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:33.892416 5709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39973
I20260927 21:12:33.892468 5775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39973 every 8 connection(s)
I20260927 21:12:33.892804 5709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:33.893546 5776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:33.896279 5776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88: Bootstrap starting.
I20260927 21:12:33.896883 5776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:33.897152 5776 log.cc:824] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:33.897984 5776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88: No bootstrap required, opened a new log
I20260927 21:12:33.899653 5776 raft_consensus.cc:348] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15ca94a24ec4fa8901f0b68a901ed88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } }
I20260927 21:12:33.899839 5776 raft_consensus.cc:374] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:33.899863 5776 raft_consensus.cc:729] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15ca94a24ec4fa8901f0b68a901ed88, State: Initialized, Role: FOLLOWER
I20260927 21:12:33.900004 5776 consensus_queue.cc:260] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15ca94a24ec4fa8901f0b68a901ed88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } }
I20260927 21:12:33.900068 5776 raft_consensus.cc:388] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:33.900090 5776 raft_consensus.cc:482] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:33.900136 5776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:33.900735 5776 raft_consensus.cc:504] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15ca94a24ec4fa8901f0b68a901ed88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } }
I20260927 21:12:33.900842 5776 leader_election.cc:302] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b15ca94a24ec4fa8901f0b68a901ed88; no voters:
I20260927 21:12:33.901016 5776 leader_election.cc:288] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:33.901096 5779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:33.901309 5776 sys_catalog.cc:564] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:33.901333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 5709
I20260927 21:12:33.901345 5779 raft_consensus.cc:686] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [term 1 LEADER]: Becoming Leader. State: Replica: b15ca94a24ec4fa8901f0b68a901ed88, State: Running, Role: LEADER
I20260927 21:12:33.901427 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:33.901496 5779 consensus_queue.cc:237] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15ca94a24ec4fa8901f0b68a901ed88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } }
I20260927 21:12:33.901592 392 external_mini_cluster.cc:1428] Setting key 519ecf052aadbded5e85db1b0f9a68ef
I20260927 21:12:33.902037 5780 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b15ca94a24ec4fa8901f0b68a901ed88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15ca94a24ec4fa8901f0b68a901ed88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } } }
I20260927 21:12:33.902114 5780 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:33.902333 5781 sys_catalog.cc:455] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b15ca94a24ec4fa8901f0b68a901ed88. Latest consensus state: current_term: 1 leader_uuid: "b15ca94a24ec4fa8901f0b68a901ed88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b15ca94a24ec4fa8901f0b68a901ed88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } } }
I20260927 21:12:33.902622 5789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:33.902674 5781 sys_catalog.cc:458] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:33.903405 5789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:33.905061 5789 catalog_manager.cc:1347] Generated new cluster ID: f9a63ed22dbf44558a570eb719661cdc
I20260927 21:12:33.905113 5789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:33.907989 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.902285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.0.98.62:39973)
0927 21:12:33.902515 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:33.902525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:33.902720 (+ 195us) server_negotiation.cc:407] Connection header received
0927 21:12:33.902983 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:33.902992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:33.903069 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:33.903248 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:33.903549 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.904668 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.905788 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.906413 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.906526 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:33.907259 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:33.907263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:33.907263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:33.907474 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:33.907517 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:33.907532 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:33.907591 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:33.907671 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 21:12:33.907758 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260927 21:12:33.908929 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:12:33.925029 5789 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:33.925820 5789 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:33.934509 5789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b15ca94a24ec4fa8901f0b68a901ed88: Generated new TSK 0
I20260927 21:12:33.934741 5789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:12:33.952740 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:33.948680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.0.98.62:39973)
0927 21:12:33.948735 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:33.948737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:33.948816 (+ 79us) server_negotiation.cc:407] Connection header received
0927 21:12:33.948905 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:33.948905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:33.948916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:33.948984 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:12:33.949118 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.949660 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.950881 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.951092 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.951168 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:33.951486 (+ 318us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:12:33.952502 (+ 1016us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:12:33.952649 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38518: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 21:12:33.952808 5792 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38518: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 21:12:33.952840 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:33.948572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39973 (local address 127.0.0.1:38518)
0927 21:12:33.948737 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:33.948750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:33.948827 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:33.948995 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:33.948996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:33.948998 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:12:33.949060 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:33.949062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.949760 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.949761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:33.950750 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:33.950753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:33.951179 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:33.951245 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:33.951324 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:33.951334 (+ 10us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:12:33.952612 (+ 1278us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:12:33.952738 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39973: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34339/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":125,"threads_started":1}
I20260927 21:12:33.953091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 5709
E20260927 21:12:33.963474 5806 webserver.cc:533] Webserver: sslize failed:
E20260927 21:12:33.963568 5806 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T21:12:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (221 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T21:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:33Z Disabled control of system clock
I20260927 21:12:33.996999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38229
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38229
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/testchainca.cert with env {}
W20260927 21:12:34.077656 5810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:34.077886 5810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:34.077910 5810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:34.079178 5810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:34.079232 5810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:34.079244 5810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:34.079257 5810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:34.079267 5810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:12:34.079278 5810 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:12:34.079289 5810 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 21:12:34.080978 5810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:34.081223 5810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:34.081566 5810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:34.086565 5816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:34.086683 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:34.086565 5818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:34.087154 5810 server_base.cc:1034] running on GCE node
I20260927 21:12:34.087440 5810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:34.087752 5810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:34.088903 5810 hybrid_clock.cc:648] HybridClock initialized: now 1790543554088874 us; error 49 us; skew 500 ppm
I20260927 21:12:34.090271 5810 webserver.cc:460] Webserver started at http://127.0.98.62:32793/ using document root <none> and password file <none>
I20260927 21:12:34.090497 5810 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:34.090547 5810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:34.090695 5810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:34.091766 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "0949ff49245b41dc8666ec6bb5909bb7"
format_stamp: "Formatted at 2026-09-27 21:12:34 on dist-test-slave-r9x1"
server_key: "d3311e46ca65d3bd829ac6312c485dfc"
server_key_iv: "515b35f3f783599c7522840b9370136f"
server_key_version: "encryptionkey@0"
I20260927 21:12:34.092072 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "0949ff49245b41dc8666ec6bb5909bb7"
format_stamp: "Formatted at 2026-09-27 21:12:34 on dist-test-slave-r9x1"
server_key: "d3311e46ca65d3bd829ac6312c485dfc"
server_key_iv: "515b35f3f783599c7522840b9370136f"
server_key_version: "encryptionkey@0"
I20260927 21:12:34.093307 5810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:34.094039 5824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:34.094199 5810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:12:34.094285 5810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "0949ff49245b41dc8666ec6bb5909bb7"
format_stamp: "Formatted at 2026-09-27 21:12:34 on dist-test-slave-r9x1"
server_key: "d3311e46ca65d3bd829ac6312c485dfc"
server_key_iv: "515b35f3f783599c7522840b9370136f"
server_key_version: "encryptionkey@0"
I20260927 21:12:34.094353 5810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:34.101891 5810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:34.102226 5810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:34.102375 5810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:34.107167 5810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38229
I20260927 21:12:34.107208 5876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38229 every 8 connection(s)
I20260927 21:12:34.107558 5810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:34.108534 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:34.111063 5877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7: Bootstrap starting.
I20260927 21:12:34.111388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 5810
I20260927 21:12:34.111495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:34.111675 392 external_mini_cluster.cc:1428] Setting key f91b346ce04ff997a8b0ec1b066277d6
I20260927 21:12:34.111824 5877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:34.112105 5877 log.cc:824] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:34.112792 5877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7: No bootstrap required, opened a new log
I20260927 21:12:34.114799 5877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0949ff49245b41dc8666ec6bb5909bb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38229 } }
I20260927 21:12:34.114961 5877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:34.114996 5877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0949ff49245b41dc8666ec6bb5909bb7, State: Initialized, Role: FOLLOWER
I20260927 21:12:34.115130 5877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0949ff49245b41dc8666ec6bb5909bb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38229 } }
I20260927 21:12:34.115217 5877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:34.115262 5877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:34.115312 5877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:34.115916 5877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0949ff49245b41dc8666ec6bb5909bb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38229 } }
I20260927 21:12:34.116021 5877 leader_election.cc:302] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0949ff49245b41dc8666ec6bb5909bb7; no voters:
I20260927 21:12:34.116204 5877 leader_election.cc:288] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:34.116236 5882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:34.116422 5882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [term 1 LEADER]: Becoming Leader. State: Replica: 0949ff49245b41dc8666ec6bb5909bb7, State: Running, Role: LEADER
I20260927 21:12:34.116485 5877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:34.116545 5882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0949ff49245b41dc8666ec6bb5909bb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38229 } }
I20260927 21:12:34.116999 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0949ff49245b41dc8666ec6bb5909bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0949ff49245b41dc8666ec6bb5909bb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38229 } } }
I20260927 21:12:34.117097 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:34.117197 5882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0949ff49245b41dc8666ec6bb5909bb7. Latest consensus state: current_term: 1 leader_uuid: "0949ff49245b41dc8666ec6bb5909bb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0949ff49245b41dc8666ec6bb5909bb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38229 } } }
I20260927 21:12:34.117311 5882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:34.117390 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:34.112358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.0.98.62:38229)
0927 21:12:34.112553 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:34.112561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:34.112586 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:12:34.113099 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:34.113107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:34.113171 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:34.113391 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:34.113526 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:34.114414 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:34.115205 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:34.115713 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:34.115832 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:34.116603 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:34.116607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:34.116608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:12:34.116828 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:34.116877 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:12:34.116893 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:34.116951 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:34.117025 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:12:34.117113 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260927 21:12:34.117967 5889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:34.118345 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:12:34.118471 5889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:34.120098 5889 catalog_manager.cc:1347] Generated new cluster ID: 35653cd7cf6f41519d24f1d15c989e2e
I20260927 21:12:34.120163 5889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:34.129835 5889 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:34.130379 5889 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:34.139381 5889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0949ff49245b41dc8666ec6bb5909bb7: Generated new TSK 0
I20260927 21:12:34.139583 5889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:12:34.608611 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:34.151891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.0.98.62:38229)
0927 21:12:34.151941 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:34.151943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:34.152011 (+ 68us) server_negotiation.cc:407] Connection header received
0927 21:12:34.152086 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:34.152087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:34.152100 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:34.152188 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:12:34.152302 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:34.153113 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:34.153978 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:34.154269 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:34.154343 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:34.154594 (+ 251us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:12:34.608302 (+453708us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:12:34.608515 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 21:12:34.608706 5881 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 21:12:34.608708 5906 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:34.151782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38229 (local address 127.0.0.1:40222)
0927 21:12:34.151942 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:34.151953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:34.152011 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:34.152195 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:34.152196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:34.152197 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:12:34.152251 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:34.152252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:34.153234 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:34.153235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:34.153890 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:34.153893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:34.154406 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:34.154414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:34.154474 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:34.154480 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:12:34.608473 (+453993us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:12:34.608620 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260927 21:12:34.609058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 5810
2026-09-27T21:12:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (635 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:34 dist-test-slave-r9x1 krb5kdc[5918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:34 dist-test-slave-r9x1 krb5kdc[5918](info): set up 2 sockets
Sep 27 21:12:34 dist-test-slave-r9x1 krb5kdc[5918](info): commencing operation
krb5kdc: starting...
W20260927 21:12:35.753576 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:35 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:35.871582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37961
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:35.955427 5931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:35.955693 5931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:35.955730 5931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:35.957033 5931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:35.957085 5931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:35.957104 5931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:35.957121 5931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:35.957136 5931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:35.958856 5931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:35.959139 5931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:35.959365 5931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:35.964421 5939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:35.964573 5931 server_base.cc:1034] running on GCE node
W20260927 21:12:35.964437 5937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:35.964712 5936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:35.964987 5931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:35.965288 5931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:35.966475 5931 hybrid_clock.cc:648] HybridClock initialized: now 1790543555966453 us; error 40 us; skew 500 ppm
Sep 27 21:12:35 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:35.969167 5931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:35.969669 5931 webserver.cc:460] Webserver started at http://127.0.98.62:40605/ using document root <none> and password file <none>
I20260927 21:12:35.969897 5931 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:35.969949 5931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:35.970067 5931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:35.971184 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "6b13d53485ae411880ca6b7ede308e4a"
format_stamp: "Formatted at 2026-09-27 21:12:35 on dist-test-slave-r9x1"
server_key: "fefe3d57afae49e29620b3ab9a6f5fb8"
server_key_iv: "9426592505439ad00d91617a22a0a49d"
server_key_version: "encryptionkey@0"
I20260927 21:12:35.971513 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b13d53485ae411880ca6b7ede308e4a"
format_stamp: "Formatted at 2026-09-27 21:12:35 on dist-test-slave-r9x1"
server_key: "fefe3d57afae49e29620b3ab9a6f5fb8"
server_key_iv: "9426592505439ad00d91617a22a0a49d"
server_key_version: "encryptionkey@0"
I20260927 21:12:35.972834 5931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:35.973567 5946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:35.973732 5931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:35.973822 5931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "6b13d53485ae411880ca6b7ede308e4a"
format_stamp: "Formatted at 2026-09-27 21:12:35 on dist-test-slave-r9x1"
server_key: "fefe3d57afae49e29620b3ab9a6f5fb8"
server_key_iv: "9426592505439ad00d91617a22a0a49d"
server_key_version: "encryptionkey@0"
I20260927 21:12:35.973888 5931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:35.982309 5931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:35.984450 5931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:35.984592 5931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:35.988471 5931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37961
I20260927 21:12:35.988533 5998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37961 every 8 connection(s)
I20260927 21:12:35.989022 5931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:35.989725 5999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:35.992060 5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a: Bootstrap starting.
I20260927 21:12:35.992645 5999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:35.992919 5999 log.cc:824] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:35.993780 5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a: No bootstrap required, opened a new log
I20260927 21:12:35.995438 5999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b13d53485ae411880ca6b7ede308e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } }
I20260927 21:12:35.995599 5999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:35.995622 5999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b13d53485ae411880ca6b7ede308e4a, State: Initialized, Role: FOLLOWER
I20260927 21:12:35.995762 5999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b13d53485ae411880ca6b7ede308e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } }
I20260927 21:12:35.995841 5999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:35.995862 5999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:35.995908 5999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:35.996570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 5931
I20260927 21:12:35.996652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:35.996812 392 external_mini_cluster.cc:1428] Setting key d4d4177d858463c8bc0a9981b0457592
I20260927 21:12:35.996898 5999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b13d53485ae411880ca6b7ede308e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } }
I20260927 21:12:35.997015 5999 leader_election.cc:302] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b13d53485ae411880ca6b7ede308e4a; no voters:
I20260927 21:12:35.997207 5999 leader_election.cc:288] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:35.997372 6002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:35.997531 6002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [term 1 LEADER]: Becoming Leader. State: Replica: 6b13d53485ae411880ca6b7ede308e4a, State: Running, Role: LEADER
I20260927 21:12:35.997539 5999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:35.997630 6002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b13d53485ae411880ca6b7ede308e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } }
I20260927 21:12:35.997952 6004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b13d53485ae411880ca6b7ede308e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b13d53485ae411880ca6b7ede308e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } } }
I20260927 21:12:35.997977 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b13d53485ae411880ca6b7ede308e4a. Latest consensus state: current_term: 1 leader_uuid: "6b13d53485ae411880ca6b7ede308e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b13d53485ae411880ca6b7ede308e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } } }
I20260927 21:12:35.998047 6004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:35.998064 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:35.998605 6010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:35.999147 6010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:36.001748 6010 catalog_manager.cc:1347] Generated new cluster ID: 298a4f3e6fc74ca7a107e41dfe163d2c
I20260927 21:12:36.001809 6010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:12:35 dist-test-slave-r9x1 krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:36.009897 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:35.997397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.0.98.62:37961)
0927 21:12:35.997920 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:35.997931 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:35.997971 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:12:35.998095 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:35.998103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:35.998169 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:35.998317 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:35.999054 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.000402 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.001508 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.002176 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.002284 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.005391 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:36.005396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:36.005397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:36.005412 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:36.008002 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.008469 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.008472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.008473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.008608 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.008813 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.008814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.008814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.009132 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:36.009316 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:36.009494 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 21:12:36.009644 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":81,"threads_started":1}
I20260927 21:12:36.019757 6010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:36.020531 6010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:36.027618 6010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b13d53485ae411880ca6b7ede308e4a: Generated new TSK 0
I20260927 21:12:36.027868 6010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:36.056816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:36.146258 6025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:36.146499 6025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:36.146535 6025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:36.147850 6025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:36.147907 6025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:36.147946 6025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:36.149644 6025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:36.149919 6025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:36.150142 6025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:36.155511 6033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:36.155534 6030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:36.155649 6031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:36.155844 6025 server_base.cc:1034] running on GCE node
I20260927 21:12:36.156044 6025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:36.156296 6025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:36.157469 6025 hybrid_clock.cc:648] HybridClock initialized: now 1790543556157454 us; error 28 us; skew 500 ppm
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.160095 6025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:36.160620 6025 webserver.cc:460] Webserver started at http://127.0.98.1:45741/ using document root <none> and password file <none>
I20260927 21:12:36.160848 6025 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:36.160900 6025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:36.161031 6025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:36.162190 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "7dea2485e9db499eb125a161fb70d961"
format_stamp: "Formatted at 2026-09-27 21:12:36 on dist-test-slave-r9x1"
server_key: "aab3a9f8452acab832cd91b4256fb136"
server_key_iv: "fdcd4fc3d9d8a5d9bdd96c79630ff599"
server_key_version: "encryptionkey@0"
I20260927 21:12:36.162564 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7dea2485e9db499eb125a161fb70d961"
format_stamp: "Formatted at 2026-09-27 21:12:36 on dist-test-slave-r9x1"
server_key: "aab3a9f8452acab832cd91b4256fb136"
server_key_iv: "fdcd4fc3d9d8a5d9bdd96c79630ff599"
server_key_version: "encryptionkey@0"
I20260927 21:12:36.163903 6025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:12:36.164736 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:36.164915 6025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:36.165002 6025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "7dea2485e9db499eb125a161fb70d961"
format_stamp: "Formatted at 2026-09-27 21:12:36 on dist-test-slave-r9x1"
server_key: "aab3a9f8452acab832cd91b4256fb136"
server_key_iv: "fdcd4fc3d9d8a5d9bdd96c79630ff599"
server_key_version: "encryptionkey@0"
I20260927 21:12:36.165076 6025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:36.184458 6025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:36.187057 6025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:36.187211 6025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:36.187453 6025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:36.187779 6025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:36.187813 6025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:36.187844 6025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:36.187862 6025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:36.193590 6025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41191
I20260927 21:12:36.193675 6153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41191 every 8 connection(s)
I20260927 21:12:36.193979 6025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:36.203799 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:36.194753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38235 (local address 127.0.98.62:37961)
0927 21:12:36.194816 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:36.194818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:36.195020 (+ 202us) server_negotiation.cc:407] Connection header received
0927 21:12:36.196042 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:36.196043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:36.196057 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:36.196128 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:36.197061 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.197692 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.198889 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.199100 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.199165 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.201456 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:36.201457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:36.201458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:36.201466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:36.202594 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.202997 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.202998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.202998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.203045 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.203243 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.203244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.203244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.203353 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:36.203402 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:36.203641 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 21:12:36.203728 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:12:36.204030 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:36.194723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37961 (local address 127.0.98.1:38235)
0927 21:12:36.194887 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:36.194935 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:36.195891 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:36.196243 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:36.196247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:36.196651 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:36.196935 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:36.196942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.197793 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.197794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:36.198792 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:36.198796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.199193 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.199194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:36.199327 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.199846 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:36.199856 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:36.201301 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:36.202679 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:36.202687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:36.202689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:36.202936 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:36.203091 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:36.203091 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:36.203092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:36.203197 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:36.203405 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:36.203411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:36.203538 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 21:12:36.203624 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:12:36.203761 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260927 21:12:36.204061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 6025
I20260927 21:12:36.204156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:36.204367 392 external_mini_cluster.cc:1428] Setting key 809983d26f00e09218e7bb9e0f459b1c
I20260927 21:12:36.204696 6154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37961
I20260927 21:12:36.204802 6154 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:36.205061 6154 heartbeater.cc:511] Master 127.0.98.62:37961 requested a full tablet report, sending...
I20260927 21:12:36.205716 5963 ts_manager.cc:194] Registered new tserver with Master: 7dea2485e9db499eb125a161fb70d961 (127.0.98.1:41191)
I20260927 21:12:36.206586 5963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38235
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:36.233981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:36.319797 6159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:36.320026 6159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:36.320058 6159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:36.321380 6159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:36.321434 6159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:36.321475 6159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:36.323102 6159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:36.323371 6159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:36.323597 6159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:36.329001 6164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:36.328981 6167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:36.329037 6159 server_base.cc:1034] running on GCE node
W20260927 21:12:36.328980 6165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:36.329471 6159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:36.329741 6159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:36.330947 6159 hybrid_clock.cc:648] HybridClock initialized: now 1790543556330935 us; error 39 us; skew 500 ppm
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.333505 6159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:36.334008 6159 webserver.cc:460] Webserver started at http://127.0.98.2:36957/ using document root <none> and password file <none>
I20260927 21:12:36.334224 6159 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:36.334276 6159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:36.334404 6159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:36.335485 6159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd23996a9d734be893c894de5128061b"
format_stamp: "Formatted at 2026-09-27 21:12:36 on dist-test-slave-r9x1"
server_key: "ef4b965a014b183ffded9bf42ad7f41c"
server_key_iv: "d1eb2ac9f923eee8d33ee27a55a9a2c7"
server_key_version: "encryptionkey@0"
I20260927 21:12:36.335829 6159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd23996a9d734be893c894de5128061b"
format_stamp: "Formatted at 2026-09-27 21:12:36 on dist-test-slave-r9x1"
server_key: "ef4b965a014b183ffded9bf42ad7f41c"
server_key_iv: "d1eb2ac9f923eee8d33ee27a55a9a2c7"
server_key_version: "encryptionkey@0"
I20260927 21:12:36.337394 6159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260927 21:12:36.338251 6174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:36.338472 6159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:36.338572 6159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "cd23996a9d734be893c894de5128061b"
format_stamp: "Formatted at 2026-09-27 21:12:36 on dist-test-slave-r9x1"
server_key: "ef4b965a014b183ffded9bf42ad7f41c"
server_key_iv: "d1eb2ac9f923eee8d33ee27a55a9a2c7"
server_key_version: "encryptionkey@0"
I20260927 21:12:36.338652 6159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:36.362458 6159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:36.365003 6159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:36.365157 6159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:36.365429 6159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:36.365785 6159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:36.365823 6159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:36.365852 6159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:36.365878 6159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:36.372056 6159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41949
I20260927 21:12:36.372119 6287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41949 every 8 connection(s)
I20260927 21:12:36.372440 6159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:36.380223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 6159
I20260927 21:12:36.380329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:36.380537 392 external_mini_cluster.cc:1428] Setting key c561bc702b613215d7c7b1de00fdde36
I20260927 21:12:36.383530 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:36.373378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50157 (local address 127.0.98.62:37961)
0927 21:12:36.373434 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:36.373436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:36.373633 (+ 197us) server_negotiation.cc:407] Connection header received
0927 21:12:36.374714 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:36.374716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:36.374738 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:36.374987 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:36.375592 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.376142 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.377328 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.377532 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.377635 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.380524 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:36.380526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:36.380526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:36.380536 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:36.381966 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.382486 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.382487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.382487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.382556 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.382811 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.382812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.382813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.382953 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:36.383019 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:36.383345 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 21:12:36.383440 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:12:36.383656 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:36.373372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37961 (local address 127.0.98.2:50157)
0927 21:12:36.373520 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:36.373558 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:36.374516 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:36.374945 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:36.374949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:36.375230 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:36.375452 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:36.375457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.376248 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.376249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:36.377199 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:36.377203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.377644 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.377646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:36.377775 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.378487 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:36.378509 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:36.380409 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:36.382058 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:36.382067 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:36.382071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:36.382402 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:36.382635 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:36.382636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:36.382636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:36.382757 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:36.383040 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:36.383047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:36.383205 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 21:12:36.383307 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:12:36.383461 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260927 21:12:36.384110 6288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37961
I20260927 21:12:36.384212 6288 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:36.384407 6288 heartbeater.cc:511] Master 127.0.98.62:37961 requested a full tablet report, sending...
I20260927 21:12:36.384800 5963 ts_manager.cc:194] Registered new tserver with Master: cd23996a9d734be893c894de5128061b (127.0.98.2:41949)
I20260927 21:12:36.385207 5963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50157
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:36.407750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:36.491227 6293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:36.491468 6293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:36.491493 6293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:36.493057 6293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:36.493136 6293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:36.493167 6293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:36.495013 6293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:36.495291 6293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:36.495515 6293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:36.500762 6301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:36.500927 6293 server_base.cc:1034] running on GCE node
W20260927 21:12:36.500768 6299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:36.500768 6298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:36.501315 6293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:36.501585 6293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:36.502784 6293 hybrid_clock.cc:648] HybridClock initialized: now 1790543556502761 us; error 36 us; skew 500 ppm
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.505528 6293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:36.506047 6293 webserver.cc:460] Webserver started at http://127.0.98.3:46595/ using document root <none> and password file <none>
I20260927 21:12:36.506273 6293 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:36.506325 6293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:36.506457 6293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:36.507579 6293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "2d701a4f6a0c4fc8827bb6323cbde21c"
format_stamp: "Formatted at 2026-09-27 21:12:36 on dist-test-slave-r9x1"
server_key: "1b45c760b63fd977619cd529b2d754c6"
server_key_iv: "b164f2a2e68f1c2d576423d1f0525649"
server_key_version: "encryptionkey@0"
I20260927 21:12:36.507925 6293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2d701a4f6a0c4fc8827bb6323cbde21c"
format_stamp: "Formatted at 2026-09-27 21:12:36 on dist-test-slave-r9x1"
server_key: "1b45c760b63fd977619cd529b2d754c6"
server_key_iv: "b164f2a2e68f1c2d576423d1f0525649"
server_key_version: "encryptionkey@0"
I20260927 21:12:36.509399 6293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:36.510234 6308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:36.510433 6293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:36.510532 6293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "2d701a4f6a0c4fc8827bb6323cbde21c"
format_stamp: "Formatted at 2026-09-27 21:12:36 on dist-test-slave-r9x1"
server_key: "1b45c760b63fd977619cd529b2d754c6"
server_key_iv: "b164f2a2e68f1c2d576423d1f0525649"
server_key_version: "encryptionkey@0"
I20260927 21:12:36.510604 6293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:36.533483 6293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:36.536288 6293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:36.536448 6293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:36.536674 6293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:36.537007 6293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:36.537050 6293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:36.537078 6293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:36.537096 6293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:36.543051 6293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37975
I20260927 21:12:36.543113 6421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37975 every 8 connection(s)
I20260927 21:12:36.543440 6293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:36.553051 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:36.544233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59981 (local address 127.0.98.62:37961)
0927 21:12:36.544301 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:36.544303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:36.544485 (+ 182us) server_negotiation.cc:407] Connection header received
0927 21:12:36.545527 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:36.545529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:36.545541 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:36.545605 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:36.546245 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.546732 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.547767 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.547966 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.548029 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.550336 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:36.550338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:36.550338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:36.550346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:36.551646 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.552114 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.552115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.552116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.552206 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.552454 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.552455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.552456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.552590 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:36.552674 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:36.552913 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 21:12:36.552983 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:12:36.553247 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:36.544196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37961 (local address 127.0.98.3:59981)
0927 21:12:36.544365 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:36.544404 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:36.545390 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:36.545695 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:36.545699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:36.545970 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:36.546171 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:36.546175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.546805 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.546806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:36.547658 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:36.547661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.548086 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.548087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:36.548221 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.548754 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:36.548762 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:36.550215 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:36.551736 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:36.551743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:36.551746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:36.552012 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:36.552268 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:36.552269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:36.552269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:36.552374 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:36.552666 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:36.552670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:36.552806 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:12:36.552891 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:12:36.553018 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260927 21:12:36.553387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 6293
I20260927 21:12:36.553457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:36.553589 392 external_mini_cluster.cc:1428] Setting key 316fed4a9c15f35d4bb6ff0398fd7eec
I20260927 21:12:36.554111 6422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37961
I20260927 21:12:36.554217 6422 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:36.554399 6422 heartbeater.cc:511] Master 127.0.98.62:37961 requested a full tablet report, sending...
I20260927 21:12:36.554770 5962 ts_manager.cc:194] Registered new tserver with Master: 2d701a4f6a0c4fc8827bb6323cbde21c (127.0.98.3:37975)
I20260927 21:12:36.555160 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59981
I20260927 21:12:36.565332 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/test-admin.keytab.
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.582260 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 21:12:36.584263 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:36.583256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.0.98.1:41191)
0927 21:12:36.583425 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:36.583430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:36.583442 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:12:36.583469 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:36.583470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:36.583484 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:36.583556 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:36.584179 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53960: BlockingRecv error: recv got EOF from 127.0.0.1:53960 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
W20260927 21:12:36.584462 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5cc)
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.585568 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.586975 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:12:36.595458 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:36.587702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.0.98.1:41191)
0927 21:12:36.587767 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:36.587769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:36.587781 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:12:36.587842 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:36.587843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:36.587853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:36.587903 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:36.588334 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.589106 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.589934 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.590322 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.590392 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.592636 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:36.592647 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:36.592647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:36.592656 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:36.594182 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.594559 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.594563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.594563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.594680 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.594860 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.594861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.594862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.595067 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:36.595195 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:36.595262 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 21:12:36.595346 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 21:12:36.597342 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:36.596729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.0.98.1:41191)
0927 21:12:36.596809 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:36.596811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:36.596820 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:36.596866 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:36.596867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:36.596876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:36.596938 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:36.597213 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53976: BlockingRecv error: recv got EOF from 127.0.0.1:53976 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 21:12:36.597355 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.598412 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.599321 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:12:36.606305 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:36.599869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.0.98.1:41191)
0927 21:12:36.599958 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:36.599960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:36.599969 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:36.599992 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:36.599993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:36.600003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:36.600050 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:36.600372 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.600873 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.601627 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.601853 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.601920 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.604011 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:36.604012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:36.604012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:36.604021 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:36.605232 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.605624 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.605625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.605625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.605682 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.605830 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.605831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.605831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.605925 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:36.605993 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:36.606129 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 21:12:36.606198 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 21:12:36.607731 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:36.607276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.0.98.1:41191)
0927 21:12:36.607329 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:36.607331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:36.607342 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:12:36.607410 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:36.607411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:36.607418 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:36.607455 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:36.607683 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53986: BlockingRecv error: recv got EOF from 127.0.0.1:53986 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 21:12:36.607868 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543529727471-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.608846 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:36.609756 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:12:36.617333 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:36.610320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.0.98.1:41191)
0927 21:12:36.610378 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:36.610380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:36.610393 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:12:36.610462 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:36.610463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:36.610471 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:36.610519 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:36.610903 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.611405 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.612090 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:36.612269 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:36.612310 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:36.614442 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:36.614444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:36.614445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:36.614459 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:36.616078 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.616509 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.616511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.616511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.616579 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:36.616823 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:36.616824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:36.616824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:36.616946 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:36.617024 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:36.617128 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 21:12:36.617209 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:12:36.618105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 6025
I20260927 21:12:36.623320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 6159
I20260927 21:12:36.627385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 6293
I20260927 21:12:36.631762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 5931
2026-09-27T21:12:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1845 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[6442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[6442](info): set up 2 sockets
Sep 27 21:12:36 dist-test-slave-r9x1 krb5kdc[6442](info): commencing operation
krb5kdc: starting...
W20260927 21:12:37.597481 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:37 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 21:12:37.720257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46215
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:37.802676 6455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:37.802924 6455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:37.802948 6455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:37.804177 6455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:37.804221 6455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:37.804234 6455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:37.804246 6455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:37.804256 6455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:37.805894 6455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:37.806119 6455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:37.806315 6455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:37.811113 6460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:37.811093 6463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:37.811092 6461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:37.811121 6455 server_base.cc:1034] running on GCE node
I20260927 21:12:37.811609 6455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:37.811926 6455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:37.813107 6455 hybrid_clock.cc:648] HybridClock initialized: now 1790543557813089 us; error 30 us; skew 500 ppm
Sep 27 21:12:37 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:37.821589 6455 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 21:12:37.822062 6455 webserver.cc:460] Webserver started at http://127.0.98.62:45101/ using document root <none> and password file <none>
I20260927 21:12:37.822292 6455 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:37.822345 6455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:37.822475 6455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:37.823573 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "c96826caa6e045feaffe427ed8b5e552"
format_stamp: "Formatted at 2026-09-27 21:12:37 on dist-test-slave-r9x1"
server_key: "15248862cca6d1492a7a1fc92b68acdd"
server_key_iv: "5e09b1efc537163fb866882be3f94f0d"
server_key_version: "encryptionkey@0"
I20260927 21:12:37.823938 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "c96826caa6e045feaffe427ed8b5e552"
format_stamp: "Formatted at 2026-09-27 21:12:37 on dist-test-slave-r9x1"
server_key: "15248862cca6d1492a7a1fc92b68acdd"
server_key_iv: "5e09b1efc537163fb866882be3f94f0d"
server_key_version: "encryptionkey@0"
I20260927 21:12:37.825552 6455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:12:37.826391 6470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:37.826570 6455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:37.826659 6455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "c96826caa6e045feaffe427ed8b5e552"
format_stamp: "Formatted at 2026-09-27 21:12:37 on dist-test-slave-r9x1"
server_key: "15248862cca6d1492a7a1fc92b68acdd"
server_key_iv: "5e09b1efc537163fb866882be3f94f0d"
server_key_version: "encryptionkey@0"
I20260927 21:12:37.826725 6455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:37.843894 6455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:37.846169 6455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:37.846316 6455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:37.850495 6455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46215
I20260927 21:12:37.850554 6522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46215 every 8 connection(s)
I20260927 21:12:37.850921 6455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:37.851984 6523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:37.854385 6523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: Bootstrap starting.
I20260927 21:12:37.854985 6523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:37.855255 6523 log.cc:824] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:37.855454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 6455
I20260927 21:12:37.855557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:37.855717 392 external_mini_cluster.cc:1428] Setting key 3f0ea248e68cfb63005035e3014286f7
I20260927 21:12:37.856027 6523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: No bootstrap required, opened a new log
I20260927 21:12:37.858127 6523 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260927 21:12:37.858317 6523 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:37.858356 6523 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96826caa6e045feaffe427ed8b5e552, State: Initialized, Role: FOLLOWER
I20260927 21:12:37.858491 6523 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260927 21:12:37.858575 6523 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:37.858613 6523 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:37.858655 6523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:37.859437 6523 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260927 21:12:37.859557 6523 leader_election.cc:302] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96826caa6e045feaffe427ed8b5e552; no voters:
I20260927 21:12:37.859719 6523 leader_election.cc:288] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:37.859815 6528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:37.860000 6523 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:37.860031 6528 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 1 LEADER]: Becoming Leader. State: Replica: c96826caa6e045feaffe427ed8b5e552, State: Running, Role: LEADER
I20260927 21:12:37.860216 6528 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260927 21:12:37.860688 6529 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } }
I20260927 21:12:37.860821 6529 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:37.860688 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96826caa6e045feaffe427ed8b5e552. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } }
I20260927 21:12:37.861025 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:37.861042 6537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:37.861589 6537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:37.863528 6537 catalog_manager.cc:1347] Generated new cluster ID: 9d7846f0248d4374aabf0557eae3fdcb
I20260927 21:12:37.863576 6537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:12:37 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:37.868582 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:37.856260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.0.98.62:46215)
0927 21:12:37.856478 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:37.856488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:37.856523 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:12:37.857358 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:37.857370 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:37.857442 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:37.857606 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:37.858191 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:37.859810 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:37.860944 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:37.861629 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:37.862330 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:37.864637 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:37.864641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:37.864642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:37.864654 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:37.866814 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:37.867277 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:37.867280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:37.867280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:37.867413 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:37.867626 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:37.867627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:37.867627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:37.867939 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:37.868091 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:37.868229 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:12:37.868371 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":117,"threads_started":1}
I20260927 21:12:37.871601 6537 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:37.872185 6537 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:37.876365 6537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: Generated new TSK 0
I20260927 21:12:37.876561 6537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 21:12:37.904640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:37.986234 6549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:37.986469 6549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:37.986501 6549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:37.987746 6549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:37.987792 6549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:37.987830 6549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:37.989476 6549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:37.989729 6549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:37.989960 6549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:37.994848 6555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:37.995020 6557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:37.995111 6554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:37.995153 6549 server_base.cc:1034] running on GCE node
I20260927 21:12:37.995369 6549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:37.995617 6549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:37.996791 6549 hybrid_clock.cc:648] HybridClock initialized: now 1790543557996770 us; error 38 us; skew 500 ppm
Sep 27 21:12:37 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:37.999454 6549 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 21:12:37.999940 6549 webserver.cc:460] Webserver started at http://127.0.98.1:44323/ using document root <none> and password file <none>
I20260927 21:12:38.000146 6549 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:38.000195 6549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:38.000314 6549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:38.001410 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "6958b89a34dd480383a6e9cb5c83734c"
format_stamp: "Formatted at 2026-09-27 21:12:37 on dist-test-slave-r9x1"
server_key: "8a2adc45364dace0d377645acb44c960"
server_key_iv: "057a7b7734a3c8cce9658981ce969e73"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.001752 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6958b89a34dd480383a6e9cb5c83734c"
format_stamp: "Formatted at 2026-09-27 21:12:37 on dist-test-slave-r9x1"
server_key: "8a2adc45364dace0d377645acb44c960"
server_key_iv: "057a7b7734a3c8cce9658981ce969e73"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.003099 6549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:12:38.003903 6564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.004101 6549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:38.004204 6549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "6958b89a34dd480383a6e9cb5c83734c"
format_stamp: "Formatted at 2026-09-27 21:12:37 on dist-test-slave-r9x1"
server_key: "8a2adc45364dace0d377645acb44c960"
server_key_iv: "057a7b7734a3c8cce9658981ce969e73"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.004271 6549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:38.023787 6549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:38.025944 6549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:38.026089 6549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:38.026324 6549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:38.026629 6549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:38.026664 6549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.026693 6549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:38.026716 6549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.032662 6549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35733
I20260927 21:12:38.032723 6677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35733 every 8 connection(s)
I20260927 21:12:38.033025 6549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:38.040010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 6549
I20260927 21:12:38.040096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:38.040282 392 external_mini_cluster.cc:1428] Setting key a000f66f1c6786caf95d4e70e16ee34a
I20260927 21:12:38.043218 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.033736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45603 (local address 127.0.98.62:46215)
0927 21:12:38.033868 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.033870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.034010 (+ 140us) server_negotiation.cc:407] Connection header received
0927 21:12:38.034508 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.034510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.034521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.034587 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.035640 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.036155 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.037202 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.037480 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.037539 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.039694 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.039695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.039696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.039704 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.041193 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.041609 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.041610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.041611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.041675 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.042298 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.042300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.042301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.042565 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.042670 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.043002 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.043115 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:12:38.043390 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.033702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:45603)
0927 21:12:38.033873 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.033916 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.034401 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.034694 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.034699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.035308 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.035515 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.035520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.036251 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.036252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.037106 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.037108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.037543 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.037544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.037647 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.038132 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.038142 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.039579 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.041293 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.041303 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.041306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.041542 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.041836 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.041838 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.041838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.042141 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.042685 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.042694 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.042861 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.042967 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.043142 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260927 21:12:38.044052 6678 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260927 21:12:38.044150 6678 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:38.044330 6678 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260927 21:12:38.044886 6487 ts_manager.cc:194] Registered new tserver with Master: 6958b89a34dd480383a6e9cb5c83734c (127.0.98.1:35733)
I20260927 21:12:38.045588 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45603
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 21:12:38.069054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:38.155969 6683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:38.156378 6683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:38.156411 6683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:38.158766 6683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:38.158864 6683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:38.158905 6683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:38.160926 6683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:38.161191 6683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:38.161451 6683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:38.166800 6688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:38.166996 6683 server_base.cc:1034] running on GCE node
W20260927 21:12:38.166800 6691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:38.167147 6689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:38.167373 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:38.167637 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:38.168812 6683 hybrid_clock.cc:648] HybridClock initialized: now 1790543558168726 us; error 103 us; skew 500 ppm
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:38.171622 6683 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 21:12:38.172101 6683 webserver.cc:460] Webserver started at http://127.0.98.2:44723/ using document root <none> and password file <none>
I20260927 21:12:38.172330 6683 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:38.172377 6683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:38.172499 6683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:38.173640 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "096cc339e8854015a453c98d001e6707"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "752d171c10dcec4b9fce27af39ced572"
server_key_iv: "b2bc57d9c521c472428d49c7b052bd5f"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.174008 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "096cc339e8854015a453c98d001e6707"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "752d171c10dcec4b9fce27af39ced572"
server_key_iv: "b2bc57d9c521c472428d49c7b052bd5f"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.175400 6683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:38.176211 6698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.176394 6683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:38.176486 6683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "096cc339e8854015a453c98d001e6707"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "752d171c10dcec4b9fce27af39ced572"
server_key_iv: "b2bc57d9c521c472428d49c7b052bd5f"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.176553 6683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:38.188874 6683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:38.191017 6683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:38.191138 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:38.191370 6683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:38.191680 6683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:38.191715 6683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.191743 6683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:38.191769 6683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.197839 6683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34177
I20260927 21:12:38.197962 6811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34177 every 8 connection(s)
I20260927 21:12:38.198356 6683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:12:38.204362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 6683
I20260927 21:12:38.204464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:38.204631 392 external_mini_cluster.cc:1428] Setting key 5f073d363af6c661b5e40d8513e4ff58
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:38.209537 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.199260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48997 (local address 127.0.98.62:46215)
0927 21:12:38.199318 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.199320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.199549 (+ 229us) server_negotiation.cc:407] Connection header received
0927 21:12:38.200049 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.200050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.200063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.200113 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.201096 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.201944 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.203212 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.203545 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.203643 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.206255 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.206256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.206257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.206264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.207888 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.208451 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.208452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.208452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.208508 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.208791 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.208792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.208793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.208938 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.209005 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.209389 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.209466 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:12:38.209824 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.199217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:48997)
0927 21:12:38.199410 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.199454 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.199950 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.200188 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.200192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.200789 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.200993 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.200999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.202074 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.202076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.203096 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.203100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.203692 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.203694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.203852 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.204393 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.204403 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.206156 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.208007 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.208016 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.208019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.208379 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.208567 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.208567 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.208568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.208721 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.209014 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.209020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.209216 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.209370 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.209563 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260927 21:12:38.210383 6812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260927 21:12:38.210484 6812 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:38.210708 6812 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260927 21:12:38.211138 6487 ts_manager.cc:194] Registered new tserver with Master: 096cc339e8854015a453c98d001e6707 (127.0.98.2:34177)
I20260927 21:12:38.211508 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48997
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 21:12:38.233656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:38.316726 6817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:38.316982 6817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:38.317009 6817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:38.318439 6817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:38.318496 6817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:38.318527 6817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:38.320119 6817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:38.320353 6817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:38.320551 6817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:38.325575 6823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:38.325567 6825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:38.325567 6822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:38.325785 6817 server_base.cc:1034] running on GCE node
I20260927 21:12:38.326097 6817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:38.326364 6817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:38.327540 6817 hybrid_clock.cc:648] HybridClock initialized: now 1790543558327525 us; error 29 us; skew 500 ppm
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:38.330257 6817 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 21:12:38.331247 6817 webserver.cc:460] Webserver started at http://127.0.98.3:41279/ using document root <none> and password file <none>
I20260927 21:12:38.331465 6817 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:38.331507 6817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:38.331609 6817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:38.332638 6817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "8cc9269e0ffd4c7b80d52a71e1490726"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "3f236d93a1070074dbbb5730c3ca73fe"
server_key_iv: "f1d19da9d14a091aabfb744b274a9cf9"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.332957 6817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8cc9269e0ffd4c7b80d52a71e1490726"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "3f236d93a1070074dbbb5730c3ca73fe"
server_key_iv: "f1d19da9d14a091aabfb744b274a9cf9"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.334316 6817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:38.335028 6832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.335180 6817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:12:38.335265 6817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "8cc9269e0ffd4c7b80d52a71e1490726"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "3f236d93a1070074dbbb5730c3ca73fe"
server_key_iv: "f1d19da9d14a091aabfb744b274a9cf9"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.335331 6817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:38.356842 6817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:38.358960 6817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:38.359102 6817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:38.359332 6817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:38.359622 6817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:38.359654 6817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.359684 6817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:38.359707 6817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.365670 6817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36143
I20260927 21:12:38.365707 6945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36143 every 8 connection(s)
I20260927 21:12:38.366158 6817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:38.368800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 6817
I20260927 21:12:38.368894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:38.369051 392 external_mini_cluster.cc:1428] Setting key 150947b98b2d2a5ef1917d1ae9e059d4
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:38.376288 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.366996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57891 (local address 127.0.98.62:46215)
0927 21:12:38.367057 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.367059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.367244 (+ 185us) server_negotiation.cc:407] Connection header received
0927 21:12:38.367785 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.367786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.367799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.367854 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.368907 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.369579 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.370725 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.370938 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.370996 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.373632 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.373634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.373634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.373643 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.374748 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.375238 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.375239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.375239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.375304 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.375638 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.375640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.375640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.375790 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.375855 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.376134 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.376217 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:12:38.376585 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.366930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:57891)
0927 21:12:38.367115 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.367157 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.367680 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.367931 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.367934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.368536 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.368765 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.368770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.369775 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.369776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.370644 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.370647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.371115 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.371117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.371312 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.371971 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.371982 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.373527 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.374872 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.374881 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.374883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.375145 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.375406 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.375408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.375408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.375548 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.375853 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.375859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.376004 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.376092 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.376288 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260927 21:12:38.377095 6946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260927 21:12:38.377192 6946 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:38.377406 6946 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260927 21:12:38.377780 6487 ts_manager.cc:194] Registered new tserver with Master: 8cc9269e0ffd4c7b80d52a71e1490726 (127.0.98.3:36143)
I20260927 21:12:38.378149 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57891
I20260927 21:12:38.380681 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 21:12:38.409402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45203
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46215,127.0.98.61:45203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:38.491184 6951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:38.491438 6951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:38.491461 6951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:38.492808 6951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:38.492878 6951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:38.492893 6951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:38.492905 6951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:38.492916 6951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:38.494639 6951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46215,127.0.98.61:45203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
W20260927 21:12:38.494772 6951 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:46215,127.0.98.61:45203'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 21:12:38.494948 6951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:38.495179 6951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:38.500260 6957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:38.500267 6959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:38.500267 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:38.500490 6951 server_base.cc:1034] running on GCE node
I20260927 21:12:38.500768 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:38.501065 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:38.502219 6951 hybrid_clock.cc:648] HybridClock initialized: now 1790543558502196 us; error 36 us; skew 500 ppm
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:38.504782 6951 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 21:12:38.505223 6951 webserver.cc:460] Webserver started at http://127.0.98.61:33235/ using document root <none> and password file <none>
I20260927 21:12:38.505445 6951 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:38.505496 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:38.505589 6951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:38.506634 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data/instance:
uuid: "87f53d45be014587b6578e1bfcb8dafe"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "0393d7d8d41d80f170852dac760792b9"
server_key_iv: "9b8f5a45ae9e9bb9101dd2bb209467ce"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.506968 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal/instance:
uuid: "87f53d45be014587b6578e1bfcb8dafe"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "0393d7d8d41d80f170852dac760792b9"
server_key_iv: "9b8f5a45ae9e9bb9101dd2bb209467ce"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.508339 6951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:38.509051 6966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.509294 6951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:38.509389 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal
uuid: "87f53d45be014587b6578e1bfcb8dafe"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "0393d7d8d41d80f170852dac760792b9"
server_key_iv: "9b8f5a45ae9e9bb9101dd2bb209467ce"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.509471 6951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:38.519600 6951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:38.521914 6951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:38.522085 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:38.526602 6951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45203
I20260927 21:12:38.526805 7018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45203 every 8 connection(s)
I20260927 21:12:38.527108 6951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:12:38.528179 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:38.530092 7019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:12:38.534415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 6951
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:12:38.540490 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.530827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.0.98.62:46215)
0927 21:12:38.530884 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.530886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.531128 (+ 242us) server_negotiation.cc:407] Connection header received
0927 21:12:38.531794 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.531796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.531809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.531860 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.532482 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.532994 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.534472 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.534770 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.534862 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.537256 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.537258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.537259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.537290 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.538913 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.539367 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.539368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.539369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.539432 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.539761 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.539762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.539762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.539907 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.539959 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.540304 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.540395 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:12:38.540660 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.530825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:41450)
0927 21:12:38.531008 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.531047 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.531666 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.531928 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.531931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.532199 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.532396 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.532400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.533107 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.533108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.534236 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.534245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.534893 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.534895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.535093 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.535625 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.535636 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.537165 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.539013 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.539019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.539021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.539279 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.539515 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.539516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.539516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.539674 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.539968 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.539974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.540162 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.540262 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.540427 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260927 21:12:38.541250 7019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:12:38.543587 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.535021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.0.98.61:45203)
0927 21:12:38.535201 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.535206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.535223 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:12:38.535242 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.535243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.535260 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.535338 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.536064 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.536765 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.537621 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.537958 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.538007 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.540476 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.540483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.540484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.540491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.542355 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.542774 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.542778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.542778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.542836 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.543019 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.543019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.543020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.543183 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.543357 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.543400 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.543479 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:12:38.548960 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.541540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:34936)
0927 21:12:38.541571 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.541580 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.541618 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.541930 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.541931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.542064 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.542126 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.542127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.542908 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.542909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.543657 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.543659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.544113 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.544113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.544175 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.544546 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.544550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.545730 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.548002 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.548003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.548004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.548166 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.548329 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.548329 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.548330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.548376 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.548599 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.548600 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.548642 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.548717 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.548802 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 21:12:38.548978 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.541631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.0.98.61:45203)
0927 21:12:38.541818 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.541820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.541827 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:38.541842 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.541842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.541850 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.541903 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.542190 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.542831 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.543735 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.544030 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.544095 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.545831 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.545832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.545833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.545840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.547897 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.548231 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.548231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.548232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.548282 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.548425 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.548425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.548426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.548527 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.548574 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.548745 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.548854 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260927 21:12:38.550352 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Bootstrap starting.
I20260927 21:12:38.550987 7019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:38.551220 7019 log.cc:824] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:38.552376 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: No bootstrap required, opened a new log
I20260927 21:12:38.554337 7019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } }
I20260927 21:12:38.554553 7019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:38.554579 7019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f53d45be014587b6578e1bfcb8dafe, State: Initialized, Role: FOLLOWER
I20260927 21:12:38.554733 7019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } }
I20260927 21:12:38.555154 7019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:38.555138 7026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } }
I20260927 21:12:38.555262 7026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:12:38.557473 7037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:12:38.557566 7037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:12:38.559353 6951 master_runner.cc:428] Detected that this master 87f53d45be014587b6578e1bfcb8dafe is joining an existing cluster
I20260927 21:12:38.559412 6951 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45203
I20260927 21:12:38.567770 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.560560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.0.98.62:46215)
0927 21:12:38.560637 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.560639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.560937 (+ 298us) server_negotiation.cc:407] Connection header received
0927 21:12:38.561085 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.561087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.561104 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.561175 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.561862 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.562738 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.563865 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.564086 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.564156 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.565251 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.565253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.565253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.565261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.566635 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.567033 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.567035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.567035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.567091 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.567263 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.567264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.567264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.567388 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.567445 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.567578 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.567660 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:12:38.567785 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.560351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:41456)
0927 21:12:38.560821 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.560837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.560976 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.561191 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.561192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.561615 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.561718 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.561720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.562855 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.562856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.563749 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.563752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.564177 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.564178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.564253 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.564681 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.564686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.565159 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.566714 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.566715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.566715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.566972 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.567143 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.567144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.567144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.567209 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.567441 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.567442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.567488 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.567547 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.567661 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":128,"thread_start_us":85,"threads_started":1}
I20260927 21:12:38.568333 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.560672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.0.98.61:45203)
0927 21:12:38.560731 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.560733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.560896 (+ 163us) server_negotiation.cc:407] Connection header received
0927 21:12:38.560991 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.560991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.561004 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.561075 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.561869 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.562738 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.563946 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.564269 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.564312 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.565451 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.565452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.565452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.565458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.567008 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.567396 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.567397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.567397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.567444 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.567605 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.567606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.567606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.567723 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.567761 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.568171 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.568249 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:12:38.568639 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.560569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:34940)
0927 21:12:38.560824 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.560836 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.560905 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.561298 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.561299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.561673 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.561779 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.561782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.562864 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.562866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.563838 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.563841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.564386 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.564387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.564447 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.564836 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.564841 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.565352 (+ 511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.567093 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.567093 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.567094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.567320 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.567502 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.567503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.567503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.567549 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.567809 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.567810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.568066 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.568446 (+ 380us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.568543 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":3,"thread_start_us":153,"threads_started":1}
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:12:38.579073 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.570103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.0.98.61:45203)
0927 21:12:38.570143 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.570145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.570266 (+ 121us) server_negotiation.cc:407] Connection header received
0927 21:12:38.570330 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.570331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.570340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.570404 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.570778 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.571533 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.572700 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.572912 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.572976 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.575889 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.575892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.575892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.575907 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.577785 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.578242 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.578243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.578243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.578297 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.578461 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.578461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.578462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.578610 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.578668 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.578893 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.578977 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:12:38.579104 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.570025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:34956)
0927 21:12:38.570152 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.570207 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.570272 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.570408 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.570410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.570657 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.570716 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.570718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.571636 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.571638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.572516 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.572524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.572986 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.572988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.573107 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.573686 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.573696 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.575613 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.577893 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.577907 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.577910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.578161 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.578353 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.578353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.578354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.578410 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.578668 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.578677 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.578751 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.578857 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.578997 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260927 21:12:38.579696 6487 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45203: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45203
}
attrs {
promote: true
}
}
dest_uuid: "c96826caa6e045feaffe427ed8b5e552"
cas_config_opid_index: -1
I20260927 21:12:38.580034 6487 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } }
I20260927 21:12:38.580549 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 87f53d45be014587b6578e1bfcb8dafe (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } } }
I20260927 21:12:38.580935 6993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:38.581682 7046 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45203
I20260927 21:12:38.581789 6993 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Refusing update from remote peer c96826caa6e045feaffe427ed8b5e552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 21:12:38.581864 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } } }
I20260927 21:12:38.581935 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:38.582082 6951 master.cc:537] Master@127.0.98.61:45203 shutting down...
I20260927 21:12:38.582100 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:12:38.582888 6474 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 21:12:38.583580 6474 consensus_peers.cc:487] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 -> Peer 87f53d45be014587b6578e1bfcb8dafe (127.0.98.61:45203): Couldn't send request to peer 87f53d45be014587b6578e1bfcb8dafe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:12:38.584973 6951 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:38.585062 6951 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:38.585112 6951 tablet_replica.cc:330] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: stopping tablet replica
I20260927 21:12:38.598659 6951 master.cc:552] Master@127.0.98.61:45203 shutdown complete.
I20260927 21:12:38.598731 6951 master_runner.cc:305] Clearing existing system tablet
I20260927 21:12:38.598863 6951 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:38.598909 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:38.599668 6951 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:38.600340 7051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.600497 6951 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:38.600558 6951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:12:38.600584 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal
uuid: "87f53d45be014587b6578e1bfcb8dafe"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "0393d7d8d41d80f170852dac760792b9"
server_key_iv: "9b8f5a45ae9e9bb9101dd2bb209467ce"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.601466 6951 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:38.603142 6951 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:12:38.603228 6951 log.cc:1196] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 21:12:38.603426 6951 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Deleting consensus metadata
I20260927 21:12:38.603539 6951 master_runner.cc:315] Copying system tablet from 127.0.98.62:46215
I20260927 21:12:38.604185 6951 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46215
I20260927 21:12:38.610781 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.604546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:41460)
0927 21:12:38.604720 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.604730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.604792 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.604902 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.604903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.605071 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.605131 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.605133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.605862 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.605863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.606429 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.606431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.606767 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.606768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.606827 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.607349 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.607353 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.607671 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.609638 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.609640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.609641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.609857 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.610062 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.610063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.610063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.610121 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.610367 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.610368 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.610425 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.610518 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.610649 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260927 21:12:38.610769 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.604621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.0.98.62:46215)
0927 21:12:38.604671 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.604673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.604805 (+ 132us) server_negotiation.cc:407] Connection header received
0927 21:12:38.604839 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.604841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.604851 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.604898 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.605200 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.605788 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.606506 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.606707 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.606769 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.607765 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.607766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.607767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.607775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.609493 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.609933 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.609934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.609934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.610002 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.610173 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.610174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.610175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.610310 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.610371 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.610551 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.610649 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:12:38.611498 6507 tablet_copy_service.cc:140] P c96826caa6e045feaffe427ed8b5e552: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 87f53d45be014587b6578e1bfcb8dafe ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41460)
I20260927 21:12:38.611622 6507 tablet_copy_service.cc:161] P c96826caa6e045feaffe427ed8b5e552: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 87f53d45be014587b6578e1bfcb8dafe at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41460: session id = 87f53d45be014587b6578e1bfcb8dafe-00000000000000000000000000000000
I20260927 21:12:38.612288 6507 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:12:38.612998 6951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:12:38.614446 6951 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: tablet copy: Starting download of 0 data blocks...
I20260927 21:12:38.614609 6951 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: tablet copy: Starting download of 1 WAL segments...
I20260927 21:12:38.615571 6951 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:12:38.616649 6507 tablet_copy_service.cc:342] P c96826caa6e045feaffe427ed8b5e552: Request end of tablet copy session 87f53d45be014587b6578e1bfcb8dafe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41460
I20260927 21:12:38.616726 6507 tablet_copy_service.cc:433] P c96826caa6e045feaffe427ed8b5e552: ending tablet copy session 87f53d45be014587b6578e1bfcb8dafe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 87f53d45be014587b6578e1bfcb8dafe
I20260927 21:12:38.619320 6951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:38.620558 7059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:38.620596 7061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:38.620678 6951 server_base.cc:1034] running on GCE node
W20260927 21:12:38.620535 7058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:38.620982 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:38.621100 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:38.622229 6951 hybrid_clock.cc:648] HybridClock initialized: now 1790543558622214 us; error 31 us; skew 500 ppm
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:38.623657 6951 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 21:12:38.623925 6951 webserver.cc:460] Webserver started at http://127.0.98.61:41015/ using document root <none> and password file <none>
I20260927 21:12:38.624038 6951 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:38.624069 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:38.624747 6951 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:38.625331 7068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.625464 6951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:38.625516 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal
uuid: "87f53d45be014587b6578e1bfcb8dafe"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "0393d7d8d41d80f170852dac760792b9"
server_key_iv: "9b8f5a45ae9e9bb9101dd2bb209467ce"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.625571 6951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:38.653752 6951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:38.655575 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:38.659184 6951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45203
I20260927 21:12:38.659299 7120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45203 every 8 connection(s)
I20260927 21:12:38.659543 6951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:12:38.660296 7121 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:12:38.660744 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Bootstrap starting.
I20260927 21:12:38.663064 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:12:38.663246 7121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Bootstrap complete.
I20260927 21:12:38.663429 7121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } }
I20260927 21:12:38.663503 7121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 87f53d45be014587b6578e1bfcb8dafe, State: Initialized, Role: LEARNER
I20260927 21:12:38.663530 7121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } }
I20260927 21:12:38.663728 7121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:38.663727 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } } }
I20260927 21:12:38.663838 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: This master's current role is: LEARNER
I20260927 21:12:38.665093 7135 catalog_manager.cc:1259] Loaded cluster ID: 9d7846f0248d4374aabf0557eae3fdcb
I20260927 21:12:38.665145 7135 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: loading cluster ID for follower catalog manager: success
I20260927 21:12:38.666251 7135 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: acquiring CA information for follower catalog manager: success
I20260927 21:12:38.666536 7135 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:12:38.703334 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.696171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.0.98.61:45203)
0927 21:12:38.696324 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.696325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.696332 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:38.696360 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.696360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.696373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.696453 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.697144 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.697886 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.698990 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.699204 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.699267 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.700753 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.700755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.700755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.700763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.702108 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.702520 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.702521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.702521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.702580 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.702821 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.702822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.702823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.702956 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.703043 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.703180 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.703261 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 21:12:38.732769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36033
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46215,127.0.98.61:45203,127.0.98.60:36033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:38.815120 7139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:38.815346 7139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:38.815380 7139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:38.816607 7139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:38.816658 7139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:38.816677 7139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:38.816694 7139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:38.816708 7139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:38.818382 7139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46215,127.0.98.61:45203,127.0.98.60:36033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:38.818648 7139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:38.818874 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:38.823697 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:38.823748 7139 server_base.cc:1034] running on GCE node
W20260927 21:12:38.823724 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:38.823714 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:38.824201 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:38.824530 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:38.825663 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790543558825648 us; error 30 us; skew 500 ppm
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:38.828289 7139 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 21:12:38.828784 7139 webserver.cc:460] Webserver started at http://127.0.98.60:40689/ using document root <none> and password file <none>
I20260927 21:12:38.829010 7139 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:38.829053 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:38.829170 7139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:38.830265 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data/instance:
uuid: "cb6c4a2376c34f79a4261a7429d5d901"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "9c8384a6e2a9da3f8178ca96fdfff24e"
server_key_iv: "bfc2481fb895676e7b73d44c69c9d591"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.830607 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal/instance:
uuid: "cb6c4a2376c34f79a4261a7429d5d901"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "9c8384a6e2a9da3f8178ca96fdfff24e"
server_key_iv: "bfc2481fb895676e7b73d44c69c9d591"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.831919 7139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:12:38.832679 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.832871 7139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:38.832952 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal
uuid: "cb6c4a2376c34f79a4261a7429d5d901"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "9c8384a6e2a9da3f8178ca96fdfff24e"
server_key_iv: "bfc2481fb895676e7b73d44c69c9d591"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.833019 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:38.844866 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:38.847004 7139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:38.847126 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:38.852159 7139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36033
I20260927 21:12:38.852203 7206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36033 every 8 connection(s)
I20260927 21:12:38.852624 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:12:38.853647 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:38.855161 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:12:38.857760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 7139
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 21:12:38.865309 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.855966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.0.98.62:46215)
0927 21:12:38.856057 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.856059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.856198 (+ 139us) server_negotiation.cc:407] Connection header received
0927 21:12:38.856684 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.856686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.856697 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.856788 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.857457 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.858151 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.859567 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.859902 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.859990 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.862512 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.862514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.862514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.862522 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.863919 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.864315 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.864317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.864317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.864372 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.864589 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.864590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.864590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.864729 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.864787 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.865102 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.865186 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:12:38.865492 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.855856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:41472)
0927 21:12:38.856066 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.856118 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.856587 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.856871 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.856875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.857129 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.857376 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.857381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.858254 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.858256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.859460 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.859463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.860018 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.860020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.860132 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.860654 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.860664 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.862431 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.864014 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.864022 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.864024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.864242 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.864427 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.864428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.864429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.864537 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.864786 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.864793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.864937 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.865029 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.865231 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260927 21:12:38.865985 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:12:38.866940 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.858261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.0.98.60:36033)
0927 21:12:38.858421 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.858426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.858440 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:12:38.858457 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.858458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.858474 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.858533 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.859256 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.860179 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.861130 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.861595 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.861693 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.864240 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.864246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.864247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.864254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.865833 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.866152 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.866155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.866155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.866212 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.866406 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.866407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.866407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.866538 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.866696 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.866739 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.866842 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:12:38.872195 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.866291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.0.98.61:45203)
0927 21:12:38.866356 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.866358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.866365 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:38.866388 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.866389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.866399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.866443 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.866671 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.867324 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.868009 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.868216 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.868272 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.869950 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.869951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.869951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.869961 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.871188 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.871512 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.871513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.871513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.871571 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.871793 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.871794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.871794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.871898 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.871950 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.872054 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.872129 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:12:38.872195 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.866220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:34974)
0927 21:12:38.866260 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.866269 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.866303 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.866450 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.866451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.866575 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.866611 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.866612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.867399 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.867400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.867936 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.867938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.868273 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.868273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.868331 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.868677 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.868691 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.869858 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.871267 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.871268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.871268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.871427 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.871666 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.871667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.871667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.871730 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.871945 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.871946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.871988 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.872035 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.872123 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 21:12:38.872645 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 21:12:38.878525 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.872950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.0.98.60:36033)
0927 21:12:38.872983 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.872985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.872992 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:38.873041 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.873042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.873052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.873091 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.873335 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.873805 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.874394 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.874598 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.874651 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.876276 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.876277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.876277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.876284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.877557 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.877853 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.877854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.877854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.877901 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.878051 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.878052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.878052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.878156 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.878207 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.878330 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.878418 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260927 21:12:38.878584 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.872896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.0.1:43684)
0927 21:12:38.872942 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.872952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.872992 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.873092 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.873092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.873216 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.873252 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.873253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.873866 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.873867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.874331 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.874332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.874653 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.874654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.874697 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.875083 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.875088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.876192 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.877635 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.877636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.877636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.877796 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.877950 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.877951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.877951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.878003 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.878206 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.878207 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.878243 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.878304 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.878475 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":82}
I20260927 21:12:38.880087 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: Bootstrap starting.
I20260927 21:12:38.881433 7207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:38.881698 7207 log.cc:824] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:38.882471 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: No bootstrap required, opened a new log
I20260927 21:12:38.883926 7207 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } }
I20260927 21:12:38.884071 7207 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:38.884093 7207 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c4a2376c34f79a4261a7429d5d901, State: Initialized, Role: FOLLOWER
I20260927 21:12:38.884217 7207 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } }
I20260927 21:12:38.884564 7207 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:38.884542 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } } }
I20260927 21:12:38.884619 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:12:38.886188 7223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:12:38.886261 7223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:12:38.888515 7139 master_runner.cc:428] Detected that this master cb6c4a2376c34f79a4261a7429d5d901 is joining an existing cluster
I20260927 21:12:38.888569 7139 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36033
I20260927 21:12:38.896553 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.889630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.0.98.62:46215)
0927 21:12:38.889682 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.889684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.889804 (+ 120us) server_negotiation.cc:407] Connection header received
0927 21:12:38.889904 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.889905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.889917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.889963 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.890484 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.891232 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.892322 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.892620 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.892660 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.894106 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.894108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.894108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.894116 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.895406 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.895795 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.895796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.895796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.895848 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.896041 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.896041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.896042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.896149 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.896194 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.896414 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.896487 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:12:38.896564 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.889582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:41488)
0927 21:12:38.889743 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.889753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.889830 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.890031 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.890032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.890323 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.890389 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.890391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.891330 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.891331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.892219 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.892222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.892737 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.892738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.893134 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.893515 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.893520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.893981 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.895490 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.895491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.895492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.895716 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.895906 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.895907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.895907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.895987 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.896201 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.896202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.896254 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.896327 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.896444 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 21:12:38.908470 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.898715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.0.1:43714)
0927 21:12:38.898762 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.898772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.898827 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.899112 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.899113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.899288 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.899339 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.899340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.899995 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.899996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.900831 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.900834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.901168 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.901169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.901233 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.901617 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.901622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.903023 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.907370 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.907372 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.907372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.907573 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.907797 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.907798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.907798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.907857 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.908142 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.908143 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.908196 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.908243 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.908368 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 21:12:38.909180 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.898774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.0.98.60:36033)
0927 21:12:38.898958 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.898960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.898967 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:38.899002 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.899003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.899013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.899097 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.899405 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.899926 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.900908 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.901112 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.901154 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.903144 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.903145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.903145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.903153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.907244 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.907652 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.907653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.907653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.907724 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.907925 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.907926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.907926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.908067 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.908947 (+ 880us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.908982 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.909075 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":260,"threads_started":1}
I20260927 21:12:38.909327 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.889646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.0.98.60:36033)
0927 21:12:38.889695 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.889697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.889992 (+ 295us) server_negotiation.cc:407] Connection header received
0927 21:12:38.890019 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.890019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.890029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.890084 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.890458 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.891183 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.892167 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.892446 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.892480 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.905746 (+ 13266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.905748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.905749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.905757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.908191 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.908581 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.908582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.908582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.908628 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.908820 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.908820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.908821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.908932 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.908969 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.909131 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.909208 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:12:38.909605 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.889600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.0.1:43700)
0927 21:12:38.889925 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.889933 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.889985 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.890095 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.890096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.890298 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.890377 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.890379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.891270 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.891270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.892087 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.892088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.892554 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.892555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.892607 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.893085 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.893089 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.905639 (+ 12550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.908279 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.908280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.908280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.908510 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.908701 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.908702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.908702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.908761 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.909004 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.909005 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.909038 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.909361 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.909496 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":29,"thread_start_us":75,"threads_started":1}
I20260927 21:12:38.909729 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.889838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.0.98.61:45203)
0927 21:12:38.889880 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.889882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.890280 (+ 398us) server_negotiation.cc:407] Connection header received
0927 21:12:38.890369 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.890370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.890380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.890407 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.890771 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.891487 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.892449 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.892707 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.892745 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.906616 (+ 13871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.906618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.906619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.906627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.908290 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.908716 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.908717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.908717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.908775 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.908960 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.908960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.908961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.909098 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.909156 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.909550 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.909635 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:12:38.910226 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.889795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:34980)
0927 21:12:38.890226 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.890234 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.890301 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.890469 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.890470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.890641 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.890688 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.890689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.891580 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.891581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.892374 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.892375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.892815 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.892815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.892865 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.905991 (+ 13126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.905996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.906509 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.908413 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.908414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.908415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.908621 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.908846 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.908847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.908847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.908900 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.909182 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.909182 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.909221 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.909318 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.909439 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":74,"threads_started":1}
I20260927 21:12:38.910495 6487 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36033: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36033
}
attrs {
promote: true
}
}
dest_uuid: "c96826caa6e045feaffe427ed8b5e552"
cas_config_opid_index: 5
I20260927 21:12:38.910749 6487 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } }
I20260927 21:12:38.911116 7047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cb6c4a2376c34f79a4261a7429d5d901 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.911455 7181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:38.911948 7047 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36033
I20260927 21:12:38.912020 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.912081 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:38.912261 7139 master.cc:537] Master@127.0.98.60:36033 shutting down...
I20260927 21:12:38.912359 7181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 FOLLOWER]: Refusing update from remote peer c96826caa6e045feaffe427ed8b5e552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:12:38.912649 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:12:38.913296 6472 consensus_peers.cc:487] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 -> Peer cb6c4a2376c34f79a4261a7429d5d901 (127.0.98.60:36033): Couldn't send request to peer cb6c4a2376c34f79a4261a7429d5d901. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:12:38.915686 7139 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:38.915795 7139 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:38.915849 7139 tablet_replica.cc:330] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: stopping tablet replica
I20260927 21:12:38.916246 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.911161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:34992)
0927 21:12:38.911240 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.911250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.911283 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.912734 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.912735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.912737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:38.912786 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.912787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.913753 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.913755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.914938 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.914941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.915810 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.915812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.915924 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.915948 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.916125 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.916179 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:12:38.916568 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.911208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.0.98.61:45203)
0927 21:12:38.912549 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.912552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.912562 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:38.912584 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.912585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.912607 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.912674 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:38.912852 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.913660 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.915016 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.915583 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.915656 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.915724 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.916455 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.916508 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315}
I20260927 21:12:38.917353 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 LEARNER]: Refusing update from remote peer c96826caa6e045feaffe427ed8b5e552: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:12:38.917550 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 21:12:38.918222 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cb6c4a2376c34f79a4261a7429d5d901 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.919327 7234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.919448 7234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: This master's current role is: LEARNER
I20260927 21:12:38.919533 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96826caa6e045feaffe427ed8b5e552. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: true } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.919589 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: This master's current role is: LEARNER
I20260927 21:12:38.919838 7046 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: attempting to promote NON_VOTER 87f53d45be014587b6578e1bfcb8dafe to VOTER
I20260927 21:12:38.920033 7046 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } }
I20260927 21:12:38.920423 7095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 LEARNER]: Refusing update from remote peer c96826caa6e045feaffe427ed8b5e552: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 21:12:38.920499 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.920397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.0.1:43728)
0927 21:12:38.920430 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.920469 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 21:12:38.920629 7235 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 21:12:38.920775 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.920702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.0.1:43734)
0927 21:12:38.920734 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.920755 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 21:12:38.920866 6472 consensus_peers.cc:487] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 -> Peer cb6c4a2376c34f79a4261a7429d5d901 (127.0.98.60:36033): Couldn't send request to peer cb6c4a2376c34f79a4261a7429d5d901. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:12:38.921350 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 87f53d45be014587b6578e1bfcb8dafe (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.921497 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 87f53d45be014587b6578e1bfcb8dafe (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.922196 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.922216 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.922276 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:12:38.922238 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.922320 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:38.922288 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:38.928844 7139 master.cc:552] Master@127.0.98.60:36033 shutdown complete.
I20260927 21:12:38.928901 7139 master_runner.cc:305] Clearing existing system tablet
I20260927 21:12:38.929035 7139 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:38.929071 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:38.929724 7139 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.930176 7237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.930294 7139 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:38.930363 7139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.930410 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal
uuid: "cb6c4a2376c34f79a4261a7429d5d901"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "9c8384a6e2a9da3f8178ca96fdfff24e"
server_key_iv: "bfc2481fb895676e7b73d44c69c9d591"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.931205 7139 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:38.932724 7139 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:12:38.932801 7139 log.cc:1196] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 21:12:38.932996 7139 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: Deleting consensus metadata
I20260927 21:12:38.933110 7139 master_runner.cc:315] Copying system tablet from 127.0.98.62:46215
I20260927 21:12:38.933759 7139 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46215
I20260927 21:12:38.940290 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.934127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.0.98.62:46215)
0927 21:12:38.934179 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:38.934181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:38.934228 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:12:38.934286 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:38.934287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:38.934297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:38.934367 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:38.934660 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.935439 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.936226 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.936549 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.936612 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.937661 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:38.937663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:38.937663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:38.937670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:38.939171 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.939550 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.939551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.939551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.939601 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:38.939814 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:38.939815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:38.939815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:38.939942 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:38.940006 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:38.940149 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 21:12:38.940212 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:12:38.940366 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:38.934058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:41502)
0927 21:12:38.934167 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:38.934178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:38.934240 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:38.934382 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:38.934382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:38.934541 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:38.934599 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.934601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.935546 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.935547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.936146 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:38.936148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:38.936619 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:38.936620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:38.936666 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:38.937155 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:38.937159 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:38.937580 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:38.939253 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.939254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.939255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.939487 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.939654 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:38.939655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:38.939655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:38.939719 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:38.940005 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:38.940007 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:38.940055 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:12:38.940129 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:12:38.940238 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":70,"threads_started":1}
I20260927 21:12:38.940881 6507 tablet_copy_service.cc:140] P c96826caa6e045feaffe427ed8b5e552: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cb6c4a2376c34f79a4261a7429d5d901 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41502)
I20260927 21:12:38.940946 6507 tablet_copy_service.cc:161] P c96826caa6e045feaffe427ed8b5e552: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cb6c4a2376c34f79a4261a7429d5d901 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41502: session id = cb6c4a2376c34f79a4261a7429d5d901-00000000000000000000000000000000
I20260927 21:12:38.941572 6507 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:12:38.942116 7139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:12:38.943441 7139 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: tablet copy: Starting download of 0 data blocks...
I20260927 21:12:38.943608 7139 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: tablet copy: Starting download of 1 WAL segments...
I20260927 21:12:38.944406 7139 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:12:38.945432 6507 tablet_copy_service.cc:342] P c96826caa6e045feaffe427ed8b5e552: Request end of tablet copy session cb6c4a2376c34f79a4261a7429d5d901-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41502
I20260927 21:12:38.945500 6507 tablet_copy_service.cc:433] P c96826caa6e045feaffe427ed8b5e552: ending tablet copy session cb6c4a2376c34f79a4261a7429d5d901-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cb6c4a2376c34f79a4261a7429d5d901
I20260927 21:12:38.947992 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:38.949224 7244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:38.949203 7245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:38.954157 7247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:38.954334 7139 server_base.cc:1034] running on GCE node
I20260927 21:12:38.954425 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:38.954566 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:38.955701 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790543558955704 us; error 40 us; skew 500 ppm
Sep 27 21:12:38 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543558, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:38.957053 7139 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 21:12:38.957388 7139 webserver.cc:460] Webserver started at http://127.0.98.60:34557/ using document root <none> and password file <none>
I20260927 21:12:38.957501 7139 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:38.957530 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:38.958226 7139 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:38.958750 7254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:38.958957 7139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:38.959012 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal
uuid: "cb6c4a2376c34f79a4261a7429d5d901"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "9c8384a6e2a9da3f8178ca96fdfff24e"
server_key_iv: "bfc2481fb895676e7b73d44c69c9d591"
server_key_version: "encryptionkey@0"
I20260927 21:12:38.959069 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:38.976796 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:38.978639 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:38.981734 7139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36033
I20260927 21:12:38.981900 7306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36033 every 8 connection(s)
I20260927 21:12:38.982117 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:12:38.982820 7307 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:12:38.983204 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: Bootstrap starting.
I20260927 21:12:38.985497 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:12:38.985689 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: Bootstrap complete.
I20260927 21:12:38.985862 7307 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } }
I20260927 21:12:38.985935 7307 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cb6c4a2376c34f79a4261a7429d5d901, State: Initialized, Role: LEARNER
I20260927 21:12:38.985971 7307 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } }
I20260927 21:12:38.986136 7307 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:38.986158 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:38.986300 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:12:38.987425 7321 catalog_manager.cc:1259] Loaded cluster ID: 9d7846f0248d4374aabf0557eae3fdcb
I20260927 21:12:38.987473 7321 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: loading cluster ID for follower catalog manager: success
I20260927 21:12:38.988379 7321 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: acquiring CA information for follower catalog manager: success
I20260927 21:12:38.988705 7321 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:12:39.024991 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.018841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.0.98.60:36033)
0927 21:12:39.018987 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.018988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.018996 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:39.019019 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.019019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.019029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.019083 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.019643 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.020324 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.021018 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.021309 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.021377 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.022523 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.022524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.022525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.022532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.023911 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.024217 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.024218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.024218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.024279 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.024440 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.024441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.024441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.024557 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.024644 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.024757 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.024870 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260927 21:12:39.025520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 6549
I20260927 21:12:39.030130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35733
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46215
--webserver_port=44323
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46215,127.0.98.61:45203
--tserver_master_addrs=127.0.98.62:46215,127.0.98.61:45203,127.0.98.60:36033 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:39.115279 7323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:39.115518 7323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:39.115545 7323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:39.116815 7323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:39.116869 7323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:39.116899 7323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:39.118530 7323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44323
--tserver_master_addrs=127.0.98.62:46215,127.0.98.61:45203,127.0.98.60:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:39.118778 7323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:39.119021 7323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:39.124274 7328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:39.124269 7331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:39.124691 7329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:39.125674 7323 server_base.cc:1034] running on GCE node
I20260927 21:12:39.125924 7323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:39.126180 7323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:39.127341 7323 hybrid_clock.cc:648] HybridClock initialized: now 1790543559127323 us; error 31 us; skew 500 ppm
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:39.130038 7323 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 21:12:39.130514 7323 webserver.cc:460] Webserver started at http://127.0.98.1:44323/ using document root <none> and password file <none>
I20260927 21:12:39.130726 7323 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:39.130784 7323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:39.132181 7323 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:39.132741 7338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.132917 7323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.133010 7323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "6958b89a34dd480383a6e9cb5c83734c"
format_stamp: "Formatted at 2026-09-27 21:12:37 on dist-test-slave-r9x1"
server_key: "8a2adc45364dace0d377645acb44c960"
server_key_iv: "057a7b7734a3c8cce9658981ce969e73"
server_key_version: "encryptionkey@0"
I20260927 21:12:39.133410 7323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:39.154778 7323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:39.157232 7323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:39.157415 7323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:39.157639 7323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:39.157943 7323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:39.157976 7323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.158006 7323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:39.158030 7323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.164654 7323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35733
I20260927 21:12:39.164753 7451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35733 every 8 connection(s)
I20260927 21:12:39.165129 7323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:12:39.166632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 7323
I20260927 21:12:39.166777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 6683
I20260927 21:12:39.172871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34177
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46215
--webserver_port=44723
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46215,127.0.98.61:45203
--tserver_master_addrs=127.0.98.62:46215,127.0.98.61:45203,127.0.98.60:36033 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 21:12:39.178646 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.166365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56901 (local address 127.0.98.61:45203)
0927 21:12:39.166406 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.166408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.166417 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:39.167509 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.167510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.167519 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.167563 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.168531 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.169341 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.170994 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.171290 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.171340 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.175184 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.175186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.175186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.175194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.176763 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.177469 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.177470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.177470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.177523 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.177774 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.177775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.177775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.177926 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.177964 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.178447 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.178545 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:12:39.178646 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.165897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39965 (local address 127.0.98.62:46215)
0927 21:12:39.166147 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.166149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.166201 (+ 52us) server_negotiation.cc:407] Connection header received
0927 21:12:39.167409 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.167410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.167423 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.167451 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.169448 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.169929 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.170945 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.171243 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.171305 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.174844 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.174846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.174846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.174855 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.176721 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.177427 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.177428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.177429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.177491 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.177774 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.177775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.177775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.177917 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.177963 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.178462 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.178545 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260927 21:12:39.178871 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.165904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.98.1:33921)
0927 21:12:39.166087 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.166132 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.167278 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.167702 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.167706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.168082 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:39.168306 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.168311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.171011 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.171011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.171743 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.171746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.172250 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.172250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.172326 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.173367 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:39.173372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:39.175372 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:39.176932 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.176933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.176933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.177363 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.177583 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.177584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.177584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.177702 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.178010 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:39.178017 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:39.178216 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.178322 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.178522 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260927 21:12:39.179075 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.166212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.98.1:56901)
0927 21:12:39.166350 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.166359 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.167375 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.167779 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.167779 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.168112 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:39.168458 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.168460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.169845 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.169847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.170914 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.170918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.171405 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.171406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.171559 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.172325 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:39.172336 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:39.175105 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:39.176859 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.176860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.176860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.177392 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.177588 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.177588 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.177589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.177700 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.178012 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:39.178019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:39.178205 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.178859 (+ 654us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.178975 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":299,"thread_start_us":60,"threads_started":1}
I20260927 21:12:39.179375 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.165891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33921 (local address 127.0.98.60:36033)
0927 21:12:39.165937 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.165939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.166217 (+ 278us) server_negotiation.cc:407] Connection header received
0927 21:12:39.167524 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.167525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.167541 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.167569 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.168413 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.169327 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.171836 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.172128 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.172166 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.175462 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.175463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.175464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.175473 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.176869 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.177448 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.177450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.177450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.177515 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.177775 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.177776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.177776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.177924 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.177984 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.179167 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.179273 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:12:39.179456 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45203
I20260927 21:12:39.179543 7454 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:39.179687 7452 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36033
I20260927 21:12:39.179746 7452 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:39.179752 7454 heartbeater.cc:511] Master 127.0.98.61:45203 requested a full tablet report, sending...
I20260927 21:12:39.179838 7452 heartbeater.cc:511] Master 127.0.98.60:36033 requested a full tablet report, sending...
I20260927 21:12:39.180207 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.165904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:39965)
0927 21:12:39.166087 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.166130 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.167282 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.167706 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.167707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.168159 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:39.168307 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.168312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.170062 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.170063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.170849 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.170853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.171391 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.171393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.171559 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.172432 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:39.172436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:39.174632 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:39.176839 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.176850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.176852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.177323 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.177563 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.177564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.177564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.177700 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.178012 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:39.178019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:39.178205 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.179679 (+ 1474us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.179923 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":8,"thread_start_us":93,"threads_started":1}
I20260927 21:12:39.180277 7085 ts_manager.cc:194] Registered new tserver with Master: 6958b89a34dd480383a6e9cb5c83734c (127.0.98.1:35733)
I20260927 21:12:39.180279 7271 ts_manager.cc:194] Registered new tserver with Master: 6958b89a34dd480383a6e9cb5c83734c (127.0.98.1:35733)
I20260927 21:12:39.180740 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260927 21:12:39.180791 7453 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:39.180850 7453 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260927 21:12:39.181085 6487 ts_manager.cc:194] Re-registered known tserver with Master: 6958b89a34dd480383a6e9cb5c83734c (127.0.98.1:35733)
I20260927 21:12:39.181492 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39965
W20260927 21:12:39.259001 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:39.259218 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:39.259253 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:39.260593 7459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:39.260648 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:39.260687 7459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:39.262342 7459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44723
--tserver_master_addrs=127.0.98.62:46215,127.0.98.61:45203,127.0.98.60:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:39.262600 7459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:39.262836 7459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:39.268083 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:39.268118 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:39.268081 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:39.268357 7459 server_base.cc:1034] running on GCE node
I20260927 21:12:39.268574 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:39.268842 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:39.270061 7459 hybrid_clock.cc:648] HybridClock initialized: now 1790543559270039 us; error 37 us; skew 500 ppm
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:39.272617 7459 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 21:12:39.273123 7459 webserver.cc:460] Webserver started at http://127.0.98.2:44723/ using document root <none> and password file <none>
I20260927 21:12:39.273393 7459 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:39.273478 7459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:39.275157 7459 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:39.275864 7474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.276083 7459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:39.276175 7459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "096cc339e8854015a453c98d001e6707"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "752d171c10dcec4b9fce27af39ced572"
server_key_iv: "b2bc57d9c521c472428d49c7b052bd5f"
server_key_version: "encryptionkey@0"
I20260927 21:12:39.276533 7459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:39.292147 7459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:39.294107 7459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:39.294270 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:39.294500 7459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:39.294802 7459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:39.294835 7459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.294866 7459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:39.294890 7459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.300926 7459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34177
I20260927 21:12:39.300966 7587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34177 every 8 connection(s)
I20260927 21:12:39.301338 7459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 21:12:39.309799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 7459
I20260927 21:12:39.309942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 6817
I20260927 21:12:39.312881 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.302060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36975 (local address 127.0.98.61:45203)
0927 21:12:39.302108 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.302110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.302540 (+ 430us) server_negotiation.cc:407] Connection header received
0927 21:12:39.303593 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.303594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.303605 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.303654 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.304377 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.305155 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.306146 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.306431 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.306503 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.309692 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.309694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.309694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.309703 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.311663 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.312075 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.312076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.312077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.312132 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.312319 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.312320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.312320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.312454 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.312491 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.312692 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.312782 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:12:39.312889 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.302441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33093 (local address 127.0.98.62:46215)
0927 21:12:39.302482 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.302484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.303591 (+ 1107us) server_negotiation.cc:407] Connection header received
0927 21:12:39.303632 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.303633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.303646 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.303693 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.304387 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.304891 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.306202 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.306498 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.306554 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.309151 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.309154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.309154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.309165 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.311244 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.311757 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.311758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.311758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.311819 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.312116 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.312117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.312117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.312269 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.312311 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.312693 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.312782 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:12:39.312882 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.302060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45527 (local address 127.0.98.60:36033)
0927 21:12:39.302108 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.302110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.302393 (+ 283us) server_negotiation.cc:407] Connection header received
0927 21:12:39.303454 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.303455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.303469 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.303526 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.304406 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.305155 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.306268 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.306465 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.306520 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.310152 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.310154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.310154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.310162 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.311615 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.312027 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.312028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.312029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.312086 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.312276 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.312277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.312278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.312404 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.312443 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.312691 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.312784 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:12:39.313460 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.302354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:33093)
0927 21:12:39.303519 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.303528 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.303579 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.303739 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.303740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.304083 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:39.304281 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.304286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.305005 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.305005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.306117 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.306119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.306606 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.306606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.306664 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.307441 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:39.307446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:39.309029 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:39.311342 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.311352 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.311355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.311669 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.311903 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.311904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.311904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.312047 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.312355 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:39.312363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:39.312543 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.312980 (+ 437us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.313198 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260927 21:12:39.313661 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.302075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.98.2:45527)
0927 21:12:39.302245 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.302285 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.303339 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.303623 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.303628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.304073 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:39.304280 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.304286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.305242 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.305243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.306173 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.306175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.306571 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.306571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.306656 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.307284 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:39.307295 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:39.310045 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:39.311706 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.311707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.311707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.311945 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.312149 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.312150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.312150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.312202 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.312484 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:39.312484 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:39.312555 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.313166 (+ 611us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.313422 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":49,"thread_start_us":86,"threads_started":1}
I20260927 21:12:39.314066 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.302075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.98.2:36975)
0927 21:12:39.302473 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.302484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.303354 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.303658 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.303659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.304060 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:39.304280 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.304285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.305302 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.305303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.306051 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.306053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.306518 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.306518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.306656 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.307311 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:39.307315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:39.309568 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:39.311748 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.311749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.311750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.311995 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.312198 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.312199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.312199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.312247 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.312533 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:39.312533 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:39.312566 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.313864 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.313985 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":81,"thread_start_us":87,"threads_started":1}
I20260927 21:12:39.314495 7590 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45203
I20260927 21:12:39.314620 7590 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:39.314832 7590 heartbeater.cc:511] Master 127.0.98.61:45203 requested a full tablet report, sending...
I20260927 21:12:39.315361 7085 ts_manager.cc:194] Registered new tserver with Master: 096cc339e8854015a453c98d001e6707 (127.0.98.2:34177)
I20260927 21:12:39.316663 7589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260927 21:12:39.316696 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36033
I20260927 21:12:39.316746 7588 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:39.316744 7589 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:39.316870 7588 heartbeater.cc:511] Master 127.0.98.60:36033 requested a full tablet report, sending...
I20260927 21:12:39.316870 7589 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260927 21:12:39.317198 7271 ts_manager.cc:194] Registered new tserver with Master: 096cc339e8854015a453c98d001e6707 (127.0.98.2:34177)
I20260927 21:12:39.317193 6487 ts_manager.cc:194] Re-registered known tserver with Master: 096cc339e8854015a453c98d001e6707 (127.0.98.2:34177)
I20260927 21:12:39.317588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36143
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46215
--webserver_port=41279
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46215,127.0.98.61:45203
--tserver_master_addrs=127.0.98.62:46215,127.0.98.61:45203,127.0.98.60:36033 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:12:39.317641 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33093
W20260927 21:12:39.402060 7595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:39.402295 7595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:39.402323 7595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:39.403693 7595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:39.403759 7595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:39.403791 7595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:39.405491 7595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41279
--tserver_master_addrs=127.0.98.62:46215,127.0.98.61:45203,127.0.98.60:36033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:39.405611 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.403146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43768 (local address 127.0.98.60:36033)
0927 21:12:39.403214 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.403216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.403225 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:39.403253 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.403254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.403277 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.403332 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:39.403454 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.403998 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.404888 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.405304 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.405370 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.405456 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.405498 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.405537 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:12:39.405622 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.403060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.0.1:43768)
0927 21:12:39.403140 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.403151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.403197 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.403341 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.403341 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.403343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:39.403395 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.403396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.404074 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.404075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.404797 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.404799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.405372 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.405374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.405432 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.405436 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.405483 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.405537 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 21:12:39.405783 7595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:39.405941 7281 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 21:12:39.406028 7595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:12:39.406103 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96826caa6e045feaffe427ed8b5e552. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:39.406214 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: This master's current role is: LEARNER
W20260927 21:12:39.411655 7601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:39.411758 7603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:39.411655 7600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:39.411919 7595 server_base.cc:1034] running on GCE node
I20260927 21:12:39.412297 7595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:39.412606 7595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:39.413774 7595 hybrid_clock.cc:648] HybridClock initialized: now 1790543559413758 us; error 34 us; skew 500 ppm
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:39.416517 7595 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 21:12:39.417032 7595 webserver.cc:460] Webserver started at http://127.0.98.3:41279/ using document root <none> and password file <none>
I20260927 21:12:39.417248 7595 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:39.417346 7595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:39.418881 7595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:39.419793 7610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.419958 7595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:39.420053 7595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "8cc9269e0ffd4c7b80d52a71e1490726"
format_stamp: "Formatted at 2026-09-27 21:12:38 on dist-test-slave-r9x1"
server_key: "3f236d93a1070074dbbb5730c3ca73fe"
server_key_iv: "f1d19da9d14a091aabfb744b274a9cf9"
server_key_version: "encryptionkey@0"
I20260927 21:12:39.420444 7595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:39.430832 7595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:39.432884 7595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:39.433043 7595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:39.433343 7595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:39.433646 7595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:39.433679 7595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.433708 7595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:39.433725 7595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:39.439425 7595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36143
I20260927 21:12:39.439488 7723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36143 every 8 connection(s)
I20260927 21:12:39.439834 7595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:39.442780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 7595
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 21:12:39 dist-test-slave-r9x1 krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:39.454452 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.441029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37557 (local address 127.0.98.62:46215)
0927 21:12:39.441083 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.441085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.441094 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:39.442127 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.442129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.442139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.442190 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.443033 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.443817 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.445325 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.445628 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.445664 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.451190 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.451192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.451193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.451201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.452727 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.453333 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.453335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.453335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.453401 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.453697 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.453698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.453698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.453836 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.453886 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.454268 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.454356 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:12:39.454629 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.440562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.98.3:45687)
0927 21:12:39.440737 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.440782 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.441891 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.442151 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.442152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.442570 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:39.442987 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.442990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.443954 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.443955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.445190 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.445193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.445719 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.445720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.445868 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.446576 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:39.446580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:39.449743 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:39.453364 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.453365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.453366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.453587 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.453781 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.453781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.453782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.453834 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.454083 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:39.454084 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:39.454128 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.454213 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.454385 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":112,"thread_start_us":82,"threads_started":1}
I20260927 21:12:39.454771 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.440807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35049 (local address 127.0.98.61:45203)
0927 21:12:39.440855 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.440857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.440867 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:39.442002 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.442004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.442017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.442073 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.443034 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.443809 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.445708 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.445975 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.446019 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.449875 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.449876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.449876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.449883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.453532 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.453968 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.453969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.453969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.454029 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.454225 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.454226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.454227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.454356 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.454407 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.454595 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.454670 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:12:39.455070 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.440532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45687 (local address 127.0.98.60:36033)
0927 21:12:39.440582 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.440584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.440868 (+ 284us) server_negotiation.cc:407] Connection header received
0927 21:12:39.442024 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.442026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.442037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.442066 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.443096 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.443811 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.445314 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.445598 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.445653 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.449847 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.449849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.449849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.449858 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.453256 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.453661 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.453662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.453662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.453717 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.453896 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.453897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.453897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.454015 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.454851 (+ 836us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.454899 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.454987 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:12:39.455296 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.440562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.98.3:35049)
0927 21:12:39.440736 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.440782 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.441885 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.442136 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.442139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.442573 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:39.442918 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.442925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.443952 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.443953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.445621 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.445624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.446095 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.446096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.446169 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.446716 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:39.446720 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:39.449783 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:39.453616 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.453618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.453618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.453877 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.454097 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.454098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.454098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.454157 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.454441 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:39.454441 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:39.454488 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.454892 (+ 404us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.455002 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":94,"thread_start_us":103,"threads_started":1}
I20260927 21:12:39.455585 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36033
I20260927 21:12:39.455684 7724 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:39.455904 7724 heartbeater.cc:511] Master 127.0.98.60:36033 requested a full tablet report, sending...
I20260927 21:12:39.455693 7726 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45203
I20260927 21:12:39.456090 7726 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:39.456140 7726 heartbeater.cc:511] Master 127.0.98.61:45203 requested a full tablet report, sending...
I20260927 21:12:39.456369 7271 ts_manager.cc:194] Registered new tserver with Master: 8cc9269e0ffd4c7b80d52a71e1490726 (127.0.98.3:36143)
I20260927 21:12:39.456405 7085 ts_manager.cc:194] Registered new tserver with Master: 8cc9269e0ffd4c7b80d52a71e1490726 (127.0.98.3:36143)
I20260927 21:12:39.456677 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.444380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.0.98.61:45203)
0927 21:12:39.446139 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.446142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.446149 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:39.446169 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.446169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.446179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.446236 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.447076 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.447797 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.448574 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.448854 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.448913 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.451470 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.451471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.451471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.451480 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.453193 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.455748 (+ 2555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.455749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.455749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.455808 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.456014 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.456015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.456015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.456137 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.456218 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.456507 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.456590 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1737,"thread_start_us":62,"threads_started":1}
I20260927 21:12:39.456779 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.446882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.0.98.60:36033)
0927 21:12:39.447937 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.447938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.448097 (+ 159us) server_negotiation.cc:407] Connection header received
0927 21:12:39.448119 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.448120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.448132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.448185 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.448918 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.449662 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.450672 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.450977 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.451031 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.452887 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.452889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.452889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.452897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.454575 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.455769 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.455770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.455770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.455821 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.456016 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.456017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.456018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.456137 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.456230 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.456618 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.456695 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":57,"threads_started":1}
I20260927 21:12:39.457540 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.440872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:37557)
0927 21:12:39.440996 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.441005 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.441895 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.442200 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.442201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.442572 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:39.442918 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.442925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.443952 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.443953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.445194 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.445196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.445749 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.445750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.445868 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.446506 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:39.446515 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:39.451098 (+ 4583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:39.452829 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.452843 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.452847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.453208 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.453467 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:39.453468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:39.453468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:39.453623 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:39.453927 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:39.453933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:39.454113 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.457306 (+ 3193us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.457435 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":145,"thread_start_us":63,"threads_started":1}
I20260927 21:12:39.457957 7725 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260927 21:12:39.458016 7725 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:39.458083 7725 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260927 21:12:39.458316 6487 ts_manager.cc:194] Re-registered known tserver with Master: 8cc9269e0ffd4c7b80d52a71e1490726 (127.0.98.3:36143)
I20260927 21:12:39.458709 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37557
I20260927 21:12:39.459967 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.443907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.0.98.62:46215)
0927 21:12:39.445113 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.445115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.445304 (+ 189us) server_negotiation.cc:407] Connection header received
0927 21:12:39.445326 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.445327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.445341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.445386 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.447187 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.450018 (+ 2831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.451059 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.451363 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.451429 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.457102 (+ 5673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.457104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.457104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.457112 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.458840 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.459249 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.459250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.459251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.459311 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.459539 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.459540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.459541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.459648 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.459727 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.459820 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.459888 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":63,"threads_started":1}
I20260927 21:12:39.461609 6487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:39.463349 6487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:39.464375 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: true } } }
I20260927 21:12:39.464491 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:39.465864 7742 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: attempting to promote NON_VOTER cb6c4a2376c34f79a4261a7429d5d901 to VOTER
I20260927 21:12:39.465999 7742 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } }
I20260927 21:12:39.466420 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Refusing update from remote peer c96826caa6e045feaffe427ed8b5e552: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 21:12:39.466984 7744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 21:12:39.467056 7280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 LEARNER]: Refusing update from remote peer c96826caa6e045feaffe427ed8b5e552: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 21:12:39.467231 7744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 21:12:39.467661 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, cb6c4a2376c34f79a4261a7429d5d901 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:39.467861 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, cb6c4a2376c34f79a4261a7429d5d901 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:39.468070 7280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, cb6c4a2376c34f79a4261a7429d5d901 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:39.468504 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:39.468573 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:39.468622 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:39.468664 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:39.468716 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:39.468752 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:39.469074 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:39.469154 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:12:39.469321 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c96826caa6e045feaffe427ed8b5e552" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:39.469400 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:12:39.473536 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.469805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35733 (local address 127.0.0.1:34306)
0927 21:12:39.469908 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.469917 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.469953 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.470432 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.470433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.470435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:39.470471 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.470472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.471634 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.471635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.472415 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.472416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.473290 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.473292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.473357 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.473363 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.473396 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.473446 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":31,"thread_start_us":71,"threads_started":1}
I20260927 21:12:39.473876 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.469621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34177 (local address 127.0.0.1:34192)
0927 21:12:39.469740 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.469749 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.469803 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.470352 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.470352 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.470354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:39.470387 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.470389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.471696 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.471697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.472804 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.472806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.473668 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.473670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.473729 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.473734 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.473760 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.473812 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260927 21:12:39.473956 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.469592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36143 (local address 127.0.0.1:43052)
0927 21:12:39.469638 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.469650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.469685 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.470074 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.470074 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.470076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:39.470120 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.470121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.471595 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.471596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.472742 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.472745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.473761 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.473762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.473823 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.473826 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.473854 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.473895 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 21:12:39.473974 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.469883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.0.98.1:35733)
0927 21:12:39.470268 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.470272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.470282 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:39.470315 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.470317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.470331 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.470433 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:39.470527 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.471530 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.472503 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.473150 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.473726 (+ 576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.473826 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.473854 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.473899 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":88,"threads_started":1}
I20260927 21:12:39.474273 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.470061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.0.98.2:34177)
0927 21:12:39.470224 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.470228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.470237 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:39.470268 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.470269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.470281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.470346 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:39.470534 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.471607 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.472891 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.473578 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.474033 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.474130 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.474157 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.474199 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260927 21:12:39.474351 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.469667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.0.98.3:36143)
0927 21:12:39.469913 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.469919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.469929 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:39.469975 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.469977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.469993 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.470079 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:39.470186 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.471508 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.472833 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.473660 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.474120 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.474204 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.474232 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.474279 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260927 21:12:39.474679 7386 tablet_service.cc:1467] Processing CreateTablet for tablet c37697d1caae406db7fbb7407e22ec6a (DEFAULT_TABLE table=test-table [id=345cf1ab09814ed9ac912df3426217ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:39.474978 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37697d1caae406db7fbb7407e22ec6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:39.474897 7522 tablet_service.cc:1467] Processing CreateTablet for tablet c37697d1caae406db7fbb7407e22ec6a (DEFAULT_TABLE table=test-table [id=345cf1ab09814ed9ac912df3426217ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:39.474900 7658 tablet_service.cc:1467] Processing CreateTablet for tablet c37697d1caae406db7fbb7407e22ec6a (DEFAULT_TABLE table=test-table [id=345cf1ab09814ed9ac912df3426217ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:39.475211 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37697d1caae406db7fbb7407e22ec6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:39.475211 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37697d1caae406db7fbb7407e22ec6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:39.477941 7758 tablet_bootstrap.cc:492] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: Bootstrap starting.
I20260927 21:12:39.478896 7758 tablet_bootstrap.cc:654] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:39.479187 7758 log.cc:824] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:39.479480 7760 tablet_bootstrap.cc:492] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: Bootstrap starting.
I20260927 21:12:39.480046 7758 tablet_bootstrap.cc:492] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: No bootstrap required, opened a new log
I20260927 21:12:39.480123 7760 tablet_bootstrap.cc:654] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:39.480121 7758 ts_tablet_manager.cc:1397] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:39.480440 7760 log.cc:824] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:39.480736 7759 tablet_bootstrap.cc:492] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: Bootstrap starting.
I20260927 21:12:39.481163 7760 tablet_bootstrap.cc:492] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: No bootstrap required, opened a new log
I20260927 21:12:39.481228 7760 ts_tablet_manager.cc:1397] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:39.481524 7759 tablet_bootstrap.cc:654] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:39.481825 7759 log.cc:824] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:39.482192 7758 raft_consensus.cc:348] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:39.482353 7758 raft_consensus.cc:374] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:39.482393 7758 raft_consensus.cc:729] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6958b89a34dd480383a6e9cb5c83734c, State: Initialized, Role: FOLLOWER
I20260927 21:12:39.482545 7758 consensus_queue.cc:260] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:39.482661 7759 tablet_bootstrap.cc:492] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: No bootstrap required, opened a new log
I20260927 21:12:39.482728 7759 ts_tablet_manager.cc:1397] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 21:12:39.482822 7758 ts_tablet_manager.cc:1428] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:12:39.483134 7760 raft_consensus.cc:348] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:39.483268 7760 raft_consensus.cc:374] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:39.483297 7760 raft_consensus.cc:729] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096cc339e8854015a453c98d001e6707, State: Initialized, Role: FOLLOWER
I20260927 21:12:39.483423 7760 consensus_queue.cc:260] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:39.483693 7589 heartbeater.cc:503] Master 127.0.98.62:46215 was elected leader, sending a full tablet report...
I20260927 21:12:39.483752 7453 heartbeater.cc:503] Master 127.0.98.62:46215 was elected leader, sending a full tablet report...
I20260927 21:12:39.483973 7760 ts_tablet_manager.cc:1428] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:12:39.486653 7759 raft_consensus.cc:348] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:39.486891 7759 raft_consensus.cc:374] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:39.486937 7759 raft_consensus.cc:729] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc9269e0ffd4c7b80d52a71e1490726, State: Initialized, Role: FOLLOWER
I20260927 21:12:39.487062 7759 consensus_queue.cc:260] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:39.487322 7759 ts_tablet_manager.cc:1428] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260927 21:12:39.487360 7725 heartbeater.cc:503] Master 127.0.98.62:46215 was elected leader, sending a full tablet report...
W20260927 21:12:39.552320 7591 tablet.cc:2369] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:12:39.666864 7455 tablet.cc:2369] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:12:39.690642 7727 tablet.cc:2369] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:39.740868 7765 raft_consensus.cc:482] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:39.740998 7765 raft_consensus.cc:504] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:39.741364 7765 leader_election.cc:288] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cc9269e0ffd4c7b80d52a71e1490726 (127.0.98.3:36143), 6958b89a34dd480383a6e9cb5c83734c (127.0.98.1:35733)
I20260927 21:12:39.745484 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.741640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44759 (local address 127.0.98.3:36143)
0927 21:12:39.741688 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.741690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.741703 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:12:39.741807 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.741808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.741819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.741874 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:39.742094 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.742854 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.744468 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.745056 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.745123 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.745134 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.745367 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.745408 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:12:39.745484 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.741651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40903 (local address 127.0.98.1:35733)
0927 21:12:39.741699 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.741701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.741708 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:39.741770 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.741771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.741783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.741856 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:39.742096 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.742868 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.744469 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.745040 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.745123 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.745134 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.745367 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.745406 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:12:39.745499 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.741590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35733 (local address 127.0.98.2:40903)
0927 21:12:39.741653 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.741660 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.741696 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.741878 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.741879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.741880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:39.741983 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.741985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.742979 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.742981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.744345 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.744348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.745157 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.745159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.745259 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.745287 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.745357 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.745401 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":82,"spinlock_wait_cycles":768}
I20260927 21:12:39.745499 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.741574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36143 (local address 127.0.98.2:44759)
0927 21:12:39.741640 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:39.741652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:39.741696 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:39.741878 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:39.741879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:39.741880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:39.741983 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.741985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.742979 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.742981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.744345 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:39.744348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.745157 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.745158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:39.745301 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.745304 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:39.745350 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:12:39.745401 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":105}
I20260927 21:12:39.745855 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37697d1caae406db7fbb7407e22ec6a" candidate_uuid: "096cc339e8854015a453c98d001e6707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6958b89a34dd480383a6e9cb5c83734c" is_pre_election: true
I20260927 21:12:39.745868 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37697d1caae406db7fbb7407e22ec6a" candidate_uuid: "096cc339e8854015a453c98d001e6707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" is_pre_election: true
I20260927 21:12:39.746033 7678 raft_consensus.cc:2454] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 096cc339e8854015a453c98d001e6707 in term 0.
I20260927 21:12:39.746033 7406 raft_consensus.cc:2454] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 096cc339e8854015a453c98d001e6707 in term 0.
I20260927 21:12:39.746261 7475 leader_election.cc:302] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 096cc339e8854015a453c98d001e6707, 8cc9269e0ffd4c7b80d52a71e1490726; no voters:
I20260927 21:12:39.746449 7765 raft_consensus.cc:2781] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:39.746531 7765 raft_consensus.cc:482] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:39.746556 7765 raft_consensus.cc:3037] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:39.747278 7765 raft_consensus.cc:504] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:39.747424 7765 leader_election.cc:288] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [CANDIDATE]: Term 1 election: Requested vote from peers 8cc9269e0ffd4c7b80d52a71e1490726 (127.0.98.3:36143), 6958b89a34dd480383a6e9cb5c83734c (127.0.98.1:35733)
I20260927 21:12:39.747591 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37697d1caae406db7fbb7407e22ec6a" candidate_uuid: "096cc339e8854015a453c98d001e6707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6958b89a34dd480383a6e9cb5c83734c"
I20260927 21:12:39.747613 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37697d1caae406db7fbb7407e22ec6a" candidate_uuid: "096cc339e8854015a453c98d001e6707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc9269e0ffd4c7b80d52a71e1490726"
I20260927 21:12:39.747679 7406 raft_consensus.cc:3037] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:39.747681 7678 raft_consensus.cc:3037] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:39.748382 7678 raft_consensus.cc:2454] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 096cc339e8854015a453c98d001e6707 in term 1.
I20260927 21:12:39.748401 7406 raft_consensus.cc:2454] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 096cc339e8854015a453c98d001e6707 in term 1.
I20260927 21:12:39.748559 7476 leader_election.cc:302] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 096cc339e8854015a453c98d001e6707, 6958b89a34dd480383a6e9cb5c83734c; no voters:
I20260927 21:12:39.748653 7765 raft_consensus.cc:2781] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:39.748802 7765 raft_consensus.cc:686] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 1 LEADER]: Becoming Leader. State: Replica: 096cc339e8854015a453c98d001e6707, State: Running, Role: LEADER
I20260927 21:12:39.748899 7765 consensus_queue.cc:237] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:39.749748 6487 catalog_manager.cc:5495] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 reported cstate change: term changed from 0 to 1, leader changed from <none> to 096cc339e8854015a453c98d001e6707 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "096cc339e8854015a453c98d001e6707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } health_report { overall_health: UNKNOWN } } }
I20260927 21:12:39.757360 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.754992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.0.98.2:34177)
0927 21:12:39.755038 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.755041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.755050 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:39.755123 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.755124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.755148 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.755207 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:39.755339 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.755825 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.756573 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.756773 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.756831 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.756983 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:39.757093 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:39.757165 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.757192 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.757216 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:12:39.759158 7678 raft_consensus.cc:1261] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [term 1 FOLLOWER]: Refusing update from remote peer 096cc339e8854015a453c98d001e6707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:39.759157 7406 raft_consensus.cc:1261] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [term 1 FOLLOWER]: Refusing update from remote peer 096cc339e8854015a453c98d001e6707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:39.759516 7767 consensus_queue.cc:1035] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:39.759642 7765 consensus_queue.cc:1035] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:39.762588 7770 mvcc.cc:205] Tried to move back new op lower bound from 7334066420770758656 to 7334066420731932672. Current Snapshot: MvccSnapshot[applied={T|T < 7334066420770758656}]
I20260927 21:12:39.762706 7771 mvcc.cc:205] Tried to move back new op lower bound from 7334066420770758656 to 7334066420731932672. Current Snapshot: MvccSnapshot[applied={T|T < 7334066420770758656}]
I20260927 21:12:39.763226 7772 mvcc.cc:205] Tried to move back new op lower bound from 7334066420770758656 to 7334066420731932672. Current Snapshot: MvccSnapshot[applied={T|T < 7334066420770758656}]
I20260927 21:12:39.766501 6487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:39.766597 6487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:39.768028 6487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c96826caa6e045feaffe427ed8b5e552: Sending DeleteTablet for 3 replicas of tablet c37697d1caae406db7fbb7407e22ec6a
I20260927 21:12:39.768329 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet c37697d1caae406db7fbb7407e22ec6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34192
I20260927 21:12:39.768369 7386 tablet_service.cc:1514] Processing DeleteTablet for tablet c37697d1caae406db7fbb7407e22ec6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34306
I20260927 21:12:39.768369 7658 tablet_service.cc:1514] Processing DeleteTablet for tablet c37697d1caae406db7fbb7407e22ec6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43052
I20260927 21:12:39.768851 7782 tablet_replica.cc:330] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: stopping tablet replica
I20260927 21:12:39.769009 7782 raft_consensus.cc:2229] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:39.769169 7782 raft_consensus.cc:2258] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:39.769723 7780 tablet_replica.cc:330] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: stopping tablet replica
I20260927 21:12:39.769716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 6455
I20260927 21:12:39.770655 7782 ts_tablet_manager.cc:1905] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:39.771898 7781 tablet_replica.cc:330] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: stopping tablet replica
I20260927 21:12:39.771997 7781 raft_consensus.cc:2229] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:39.772082 7781 raft_consensus.cc:2258] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:39.772190 7780 raft_consensus.cc:2229] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:39.772358 7780 raft_consensus.cc:2258] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:39.772400 7782 ts_tablet_manager.cc:1918] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:39.772461 7782 log.cc:1196] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/c37697d1caae406db7fbb7407e22ec6a
I20260927 21:12:39.772714 7782 ts_tablet_manager.cc:1939] T c37697d1caae406db7fbb7407e22ec6a P 8cc9269e0ffd4c7b80d52a71e1490726: Deleting consensus metadata
I20260927 21:12:39.772850 7780 ts_tablet_manager.cc:1905] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:39.774580 7780 ts_tablet_manager.cc:1918] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:39.774598 7781 ts_tablet_manager.cc:1905] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:39.774736 7780 log.cc:1196] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/c37697d1caae406db7fbb7407e22ec6a
I20260927 21:12:39.775048 7780 ts_tablet_manager.cc:1939] T c37697d1caae406db7fbb7407e22ec6a P 096cc339e8854015a453c98d001e6707: Deleting consensus metadata
W20260927 21:12:39.776176 7476 connection.cc:535] server connection from 127.0.0.1:34192 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 21:12:39.776328 7611 connection.cc:535] server connection from 127.0.0.1:43052 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:12:39.776393 7781 ts_tablet_manager.cc:1918] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:39.776463 7781 log.cc:1196] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/c37697d1caae406db7fbb7407e22ec6a
I20260927 21:12:39.776677 7781 ts_tablet_manager.cc:1939] T c37697d1caae406db7fbb7407e22ec6a P 6958b89a34dd480383a6e9cb5c83734c: Deleting consensus metadata
W20260927 21:12:39.777091 7341 connection.cc:441] server connection from 127.0.0.1:34306 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34306 (request call id 1) could send its response
I20260927 21:12:39.786684 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.778326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.0.98.60:36033)
0927 21:12:39.778382 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.778383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.778396 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:12:39.778456 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.778457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.778469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.778539 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.779515 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.780270 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.782052 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.782333 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.782398 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.783532 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.783534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.783534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.783542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.785472 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.785889 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.785890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.785891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.785955 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.786161 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.786161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.786162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.786290 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.786380 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.786486 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.786582 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:12:39.787002 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:39.778583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.98.61:45203)
0927 21:12:39.778633 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:39.778634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:39.778728 (+ 94us) server_negotiation.cc:407] Connection header received
0927 21:12:39.778825 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:39.778826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:39.778841 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:39.778896 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:39.779492 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.780277 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.781434 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:39.781757 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:39.781800 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:39.783448 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:39.783450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:39.783451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:39.783458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:39.785505 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.785996 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.785997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.785997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.786055 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:39.786261 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:39.786263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:39.786263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:39.786465 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:39.786581 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:39.786792 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 21:12:39.786891 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:12:40.764150 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:40.763787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:49245)
0927 21:12:40.764006 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:40.764093 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":63,"threads_started":1}
W20260927 21:12:40.764627 7589 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
I20260927 21:12:40.764657 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:40.764240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:59095)
0927 21:12:40.764477 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:40.764613 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":49,"threads_started":1}
I20260927 21:12:40.764801 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:40.764343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:59679)
0927 21:12:40.764699 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:40.764764 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":52,"threads_started":1}
W20260927 21:12:40.764818 7725 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
W20260927 21:12:40.765004 7453 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
I20260927 21:12:41.268707 7793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c96826caa6e045feaffe427ed8b5e552)
I20260927 21:12:41.268848 7793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } }
I20260927 21:12:41.269013 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c96826caa6e045feaffe427ed8b5e552)
I20260927 21:12:41.269111 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } }
I20260927 21:12:41.269366 7794 leader_election.cc:288] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c96826caa6e045feaffe427ed8b5e552 (127.0.98.62:46215), 87f53d45be014587b6578e1bfcb8dafe (127.0.98.61:45203)
I20260927 21:12:41.269393 7793 leader_election.cc:288] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c96826caa6e045feaffe427ed8b5e552 (127.0.98.62:46215), cb6c4a2376c34f79a4261a7429d5d901 (127.0.98.60:36033)
I20260927 21:12:41.269659 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.269477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:54394)
0927 21:12:41.269547 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.269621 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":44,"threads_started":1}
I20260927 21:12:41.269759 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.269486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:54388)
0927 21:12:41.269682 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.269723 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":44,"threads_started":1}
W20260927 21:12:41.269793 7258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
W20260927 21:12:41.269904 7072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
I20260927 21:12:41.270488 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.270326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:54412)
0927 21:12:41.270422 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.270458 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 21:12:41.270627 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.270550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:54426)
0927 21:12:41.270582 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.270602 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 21:12:41.270712 7258 leader_election.cc:334] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c96826caa6e045feaffe427ed8b5e552 (127.0.98.62:46215): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
W20260927 21:12:41.270769 7072 leader_election.cc:334] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c96826caa6e045feaffe427ed8b5e552 (127.0.98.62:46215): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
I20260927 21:12:41.272560 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.269298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36033 (local address 127.0.0.1:39926)
0927 21:12:41.269496 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.269508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:41.269578 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:41.269876 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:41.269877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:41.269879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:41.269933 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.269935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.270817 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.270817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.271801 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.271803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.272392 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.272393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.272440 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.272444 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:41.272480 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 21:12:41.272513 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":49,"threads_started":1}
I20260927 21:12:41.272599 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.269416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.0.98.60:36033)
0927 21:12:41.269510 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:41.269513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:41.269561 (+ 48us) server_negotiation.cc:407] Connection header received
0927 21:12:41.269752 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:41.269753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:41.269764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:41.269822 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:41.270012 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.270728 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.271900 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.272288 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.272371 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.272383 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:41.272499 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 21:12:41.272537 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":47,"threads_started":1}
I20260927 21:12:41.272756 7280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87f53d45be014587b6578e1bfcb8dafe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cb6c4a2376c34f79a4261a7429d5d901" is_pre_election: true
I20260927 21:12:41.272857 7280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87f53d45be014587b6578e1bfcb8dafe in term 1.
I20260927 21:12:41.273018 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87f53d45be014587b6578e1bfcb8dafe, cb6c4a2376c34f79a4261a7429d5d901; no voters: c96826caa6e045feaffe427ed8b5e552
I20260927 21:12:41.273129 7793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 21:12:41.273164 7793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Starting leader election (detected failure of leader c96826caa6e045feaffe427ed8b5e552)
I20260927 21:12:41.273200 7793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:12:41.273202 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.269617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45203 (local address 127.0.0.1:52428)
0927 21:12:41.269953 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.269963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:41.270009 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:41.270555 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:41.270556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:41.270558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:41.270604 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.270606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.271297 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.271298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.272228 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.272230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.272968 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.272969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.273036 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.273040 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:41.273098 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:12:41.273136 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":1,"thread_start_us":46,"threads_started":1}
I20260927 21:12:41.273249 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.269672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.0.98.61:45203)
0927 21:12:41.270426 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:41.270428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:41.270435 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:41.270453 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:41.270454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:41.270463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:41.270656 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:41.270684 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.271213 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.272316 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.272813 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.272948 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.272961 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:41.273111 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 21:12:41.273151 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":38,"threads_started":1}
I20260927 21:12:41.273536 7095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb6c4a2376c34f79a4261a7429d5d901" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "87f53d45be014587b6578e1bfcb8dafe" is_pre_election: true
I20260927 21:12:41.274009 7793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } }
I20260927 21:12:41.274133 7095 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb6c4a2376c34f79a4261a7429d5d901 in current term 2: Already voted for candidate 87f53d45be014587b6578e1bfcb8dafe in this term.
I20260927 21:12:41.274140 7793 leader_election.cc:288] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [CANDIDATE]: Term 2 election: Requested vote from peers c96826caa6e045feaffe427ed8b5e552 (127.0.98.62:46215), cb6c4a2376c34f79a4261a7429d5d901 (127.0.98.60:36033)
I20260927 21:12:41.274314 7258 leader_election.cc:302] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb6c4a2376c34f79a4261a7429d5d901; no voters: 87f53d45be014587b6578e1bfcb8dafe, c96826caa6e045feaffe427ed8b5e552
I20260927 21:12:41.274433 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.274256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:54440)
0927 21:12:41.274359 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.274400 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 21:12:41.274376 7280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87f53d45be014587b6578e1bfcb8dafe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cb6c4a2376c34f79a4261a7429d5d901"
I20260927 21:12:41.274459 7794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:12:41.274785 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.274700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:54454)
0927 21:12:41.274740 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.274766 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 21:12:41.274909 7072 leader_election.cc:334] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c96826caa6e045feaffe427ed8b5e552 (127.0.98.62:46215): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
I20260927 21:12:41.275206 7794 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:12:41.275810 7280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87f53d45be014587b6578e1bfcb8dafe in term 2.
I20260927 21:12:41.275985 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 87f53d45be014587b6578e1bfcb8dafe, cb6c4a2376c34f79a4261a7429d5d901; no voters: c96826caa6e045feaffe427ed8b5e552
I20260927 21:12:41.276077 7793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:12:41.276212 7793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [term 2 LEADER]: Becoming Leader. State: Replica: 87f53d45be014587b6578e1bfcb8dafe, State: Running, Role: LEADER
I20260927 21:12:41.276271 7793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } }
I20260927 21:12:41.276564 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f53d45be014587b6578e1bfcb8dafe. Latest consensus state: current_term: 2 leader_uuid: "87f53d45be014587b6578e1bfcb8dafe" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:41.276636 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:41.276830 7805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:41.277405 7805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:41.277499 7805 catalog_manager.cc:1259] Loaded cluster ID: 9d7846f0248d4374aabf0557eae3fdcb
I20260927 21:12:41.277532 7805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:41.277683 7805 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:41.277740 7805 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Loaded TSK: 0
I20260927 21:12:41.277834 7805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:12:41.326099 7084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:41.327682 7084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:41.328749 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.328604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:54472)
0927 21:12:41.328671 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.328714 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 21:12:41.328773 7280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [term 2 FOLLOWER]: Refusing update from remote peer 87f53d45be014587b6578e1bfcb8dafe: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 21:12:41.329015 7803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 21:12:41.329125 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.329025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.0.1:54476)
0927 21:12:41.329074 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.329100 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:12:41.329192 7072 consensus_peers.cc:487] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe -> Peer c96826caa6e045feaffe427ed8b5e552 (127.0.98.62:46215): Couldn't send request to peer c96826caa6e045feaffe427ed8b5e552. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:12:41.329627 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f53d45be014587b6578e1bfcb8dafe. Latest consensus state: current_term: 2 leader_uuid: "87f53d45be014587b6578e1bfcb8dafe" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:41.329715 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:12:41.330005 7806 mvcc.cc:205] Tried to move back new op lower bound from 7334066427200528384 to 7334066426988171264. Current Snapshot: MvccSnapshot[applied={T|T < 7334066427200528384}]
I20260927 21:12:41.330190 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "87f53d45be014587b6578e1bfcb8dafe" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:41.330272 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:41.330345 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "87f53d45be014587b6578e1bfcb8dafe" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c96826caa6e045feaffe427ed8b5e552" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } peers { permanent_uuid: "87f53d45be014587b6578e1bfcb8dafe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45203 } attrs { promote: false } } peers { permanent_uuid: "cb6c4a2376c34f79a4261a7429d5d901" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36033 } attrs { promote: false } } }
I20260927 21:12:41.330454 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6c4a2376c34f79a4261a7429d5d901 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:12:41.330580 7807 mvcc.cc:205] Tried to move back new op lower bound from 7334066427200528384 to 7334066426988171264. Current Snapshot: MvccSnapshot[applied={T|T < 7334066427200528384 or (T in {7334066427200528384})}]
I20260927 21:12:41.335330 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.332372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.0.98.1:35733)
0927 21:12:41.332680 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:41.332682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:41.332689 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:41.332712 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:41.332713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:41.332752 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:41.332817 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:41.332924 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.333585 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.334528 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.334993 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.335061 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.335075 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:41.335210 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 21:12:41.335251 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":74,"threads_started":1}
I20260927 21:12:41.335383 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.332057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35733 (local address 127.0.0.1:40046)
0927 21:12:41.332176 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.332186 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:41.332236 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:41.332832 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:41.332833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:41.332835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:41.332871 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.332872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.333652 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.333653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.334425 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.334427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.335076 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.335077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.335144 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.335149 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:12:41.335197 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:12:41.335245 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260927 21:12:41.335394 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.332005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36143 (local address 127.0.0.1:51124)
0927 21:12:41.332147 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.332157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:41.332197 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:41.332462 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:41.332463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:41.332466 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:41.332529 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.332530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.333304 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.333305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.334501 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.334502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.335121 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.335122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.335188 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.335191 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:41.335237 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:12:41.335286 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":3}
I20260927 21:12:41.335552 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.332255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.0.98.3:36143)
0927 21:12:41.332353 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:41.332355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:41.332362 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:41.332382 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:41.332382 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:41.332389 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:41.332439 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:41.332588 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.333205 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.334595 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.335034 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.335108 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.335121 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:41.335423 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:12:41.335481 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":57,"threads_started":1}
I20260927 21:12:41.335821 7658 tablet_service.cc:1467] Processing CreateTablet for tablet ec7e384b588f4c678bfdad43d3ac84b8 (DEFAULT_TABLE table=test-table [id=e76d305623d4464195e350bcd55dc219]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:41.335960 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7e384b588f4c678bfdad43d3ac84b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:41.336062 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.331970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34177 (local address 127.0.0.1:39830)
0927 21:12:41.332047 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.332059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:41.332095 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:41.332400 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:41.332401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:41.332403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:41.332456 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.332457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.333412 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.333412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.334489 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.334490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.335809 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.335810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.335879 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.335883 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:41.335930 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:12:41.335975 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":3}
I20260927 21:12:41.336230 7386 tablet_service.cc:1467] Processing CreateTablet for tablet ec7e384b588f4c678bfdad43d3ac84b8 (DEFAULT_TABLE table=test-table [id=e76d305623d4464195e350bcd55dc219]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:41.336272 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.332038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.0.98.2:34177)
0927 21:12:41.332272 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:41.332274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:41.332282 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:41.332307 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:41.332308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:41.332320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:41.332515 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:41.332555 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.333337 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.335159 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.335738 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.335811 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.335825 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:41.336150 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 21:12:41.336206 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":74,"threads_started":1}
I20260927 21:12:41.336345 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7e384b588f4c678bfdad43d3ac84b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:41.336498 7522 tablet_service.cc:1467] Processing CreateTablet for tablet ec7e384b588f4c678bfdad43d3ac84b8 (DEFAULT_TABLE table=test-table [id=e76d305623d4464195e350bcd55dc219]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:41.336586 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7e384b588f4c678bfdad43d3ac84b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:41.337852 7817 tablet_bootstrap.cc:492] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c: Bootstrap starting.
I20260927 21:12:41.337915 7818 tablet_bootstrap.cc:492] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: Bootstrap starting.
I20260927 21:12:41.338012 7816 tablet_bootstrap.cc:492] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: Bootstrap starting.
I20260927 21:12:41.338313 7818 tablet_bootstrap.cc:654] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:41.338347 7817 tablet_bootstrap.cc:654] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:41.338364 7816 tablet_bootstrap.cc:654] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:41.338892 7818 tablet_bootstrap.cc:492] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: No bootstrap required, opened a new log
I20260927 21:12:41.338949 7818 ts_tablet_manager.cc:1397] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:41.338956 7816 tablet_bootstrap.cc:492] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: No bootstrap required, opened a new log
I20260927 21:12:41.338999 7816 ts_tablet_manager.cc:1397] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:41.339071 7818 raft_consensus.cc:348] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:41.339145 7818 raft_consensus.cc:374] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:41.339126 7816 raft_consensus.cc:348] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:41.339170 7818 raft_consensus.cc:729] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096cc339e8854015a453c98d001e6707, State: Initialized, Role: FOLLOWER
I20260927 21:12:41.339190 7816 raft_consensus.cc:374] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:41.339218 7816 raft_consensus.cc:729] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc9269e0ffd4c7b80d52a71e1490726, State: Initialized, Role: FOLLOWER
I20260927 21:12:41.339216 7818 consensus_queue.cc:260] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:41.339279 7816 consensus_queue.cc:260] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:41.339354 7818 ts_tablet_manager.cc:1428] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:41.339433 7816 ts_tablet_manager.cc:1428] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:41.339946 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.339660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:58657)
0927 21:12:41.339864 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.339902 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260927 21:12:41.339982 7817 tablet_bootstrap.cc:492] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c: No bootstrap required, opened a new log
I20260927 21:12:41.340036 7817 ts_tablet_manager.cc:1397] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:41.340173 7817 raft_consensus.cc:348] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:41.340230 7817 raft_consensus.cc:374] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:41.340245 7817 raft_consensus.cc:729] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6958b89a34dd480383a6e9cb5c83734c, State: Initialized, Role: FOLLOWER
I20260927 21:12:41.340288 7817 consensus_queue.cc:260] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:41.340417 7817 ts_tablet_manager.cc:1428] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:41.340749 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.340549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:50739)
0927 21:12:41.340669 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.340704 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260927 21:12:41.341003 7454 heartbeater.cc:503] Master 127.0.98.61:45203 was elected leader, sending a full tablet report...
I20260927 21:12:41.341399 7590 heartbeater.cc:503] Master 127.0.98.61:45203 was elected leader, sending a full tablet report...
I20260927 21:12:41.341403 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.339533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:37071)
0927 21:12:41.341335 (+ 1802us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.341371 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1780,"thread_start_us":90,"threads_started":1}
I20260927 21:12:41.341599 7726 heartbeater.cc:503] Master 127.0.98.61:45203 was elected leader, sending a full tablet report...
I20260927 21:12:41.341903 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.341659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:56913)
0927 21:12:41.341810 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.341837 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260927 21:12:41.466892 7823 raft_consensus.cc:482] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:41.467000 7823 raft_consensus.cc:504] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:41.467347 7823 leader_election.cc:288] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 096cc339e8854015a453c98d001e6707 (127.0.98.2:34177), 6958b89a34dd480383a6e9cb5c83734c (127.0.98.1:35733)
I20260927 21:12:41.470242 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.467712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34969 (local address 127.0.98.1:35733)
0927 21:12:41.467751 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:41.467754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:41.467856 (+ 102us) server_negotiation.cc:407] Connection header received
0927 21:12:41.467900 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:41.467901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:41.467913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:41.467954 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:41.468122 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.468597 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.469563 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.469924 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.469987 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.469999 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:41.470134 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 21:12:41.470173 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:12:41.470252 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.467636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39457 (local address 127.0.98.2:34177)
0927 21:12:41.467686 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:41.467688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:41.467700 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:12:41.467819 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:41.467820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:41.467834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:41.467885 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:41.468055 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.468530 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.469570 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.469927 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.469987 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.469999 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:41.470151 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:12:41.470187 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:12:41.470264 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.467538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34177 (local address 127.0.98.3:39457)
0927 21:12:41.467607 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.467620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:41.467700 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:41.467896 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:41.467897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:41.467899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:41.467972 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.467974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.468616 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.468617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.469475 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.469478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.470020 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.470021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.470089 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.470092 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:41.470137 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:12:41.470179 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 21:12:41.470270 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.467630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35733 (local address 127.0.98.3:34969)
0927 21:12:41.467776 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.467785 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:41.467845 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:41.467998 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:41.467999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:41.468001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:41.468054 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.468055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.468673 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.468674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.469480 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:41.469483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.469997 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.469998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:41.470070 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.470074 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:41.470119 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:12:41.470163 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":106,"threads_started":1}
I20260927 21:12:41.470505 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7e384b588f4c678bfdad43d3ac84b8" candidate_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6958b89a34dd480383a6e9cb5c83734c" is_pre_election: true
I20260927 21:12:41.470530 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7e384b588f4c678bfdad43d3ac84b8" candidate_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "096cc339e8854015a453c98d001e6707" is_pre_election: true
I20260927 21:12:41.470613 7406 raft_consensus.cc:2454] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc9269e0ffd4c7b80d52a71e1490726 in term 0.
I20260927 21:12:41.470649 7542 raft_consensus.cc:2454] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc9269e0ffd4c7b80d52a71e1490726 in term 0.
I20260927 21:12:41.470803 7612 leader_election.cc:302] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6958b89a34dd480383a6e9cb5c83734c, 8cc9269e0ffd4c7b80d52a71e1490726; no voters:
I20260927 21:12:41.470942 7823 raft_consensus.cc:2781] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:41.470983 7823 raft_consensus.cc:482] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:41.471010 7823 raft_consensus.cc:3037] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:41.471680 7823 raft_consensus.cc:504] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:41.471798 7823 leader_election.cc:288] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [CANDIDATE]: Term 1 election: Requested vote from peers 096cc339e8854015a453c98d001e6707 (127.0.98.2:34177), 6958b89a34dd480383a6e9cb5c83734c (127.0.98.1:35733)
I20260927 21:12:41.472011 7542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7e384b588f4c678bfdad43d3ac84b8" candidate_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "096cc339e8854015a453c98d001e6707"
I20260927 21:12:41.472095 7542 raft_consensus.cc:3037] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:41.472061 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7e384b588f4c678bfdad43d3ac84b8" candidate_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6958b89a34dd480383a6e9cb5c83734c"
I20260927 21:12:41.472127 7406 raft_consensus.cc:3037] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:41.472738 7542 raft_consensus.cc:2454] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc9269e0ffd4c7b80d52a71e1490726 in term 1.
I20260927 21:12:41.472749 7406 raft_consensus.cc:2454] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc9269e0ffd4c7b80d52a71e1490726 in term 1.
I20260927 21:12:41.472924 7612 leader_election.cc:302] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 096cc339e8854015a453c98d001e6707, 8cc9269e0ffd4c7b80d52a71e1490726; no voters:
I20260927 21:12:41.473027 7823 raft_consensus.cc:2781] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:41.473194 7823 raft_consensus.cc:686] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc9269e0ffd4c7b80d52a71e1490726, State: Running, Role: LEADER
I20260927 21:12:41.473286 7823 consensus_queue.cc:237] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } }
I20260927 21:12:41.473718 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.473432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:35731)
0927 21:12:41.473656 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.473689 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260927 21:12:41.474071 7085 catalog_manager.cc:5495] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cc9269e0ffd4c7b80d52a71e1490726 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc9269e0ffd4c7b80d52a71e1490726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 } health_report { overall_health: UNKNOWN } } }
I20260927 21:12:41.482462 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.480076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.0.98.3:36143)
0927 21:12:41.480119 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:41.480121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:41.480129 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:41.480161 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:41.480162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:41.480181 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:41.480231 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:41.480322 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.480825 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.481692 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:41.481902 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:41.481964 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:41.482115 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:41.482219 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:41.482271 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:41.482350 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 21:12:41.482373 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:12:41.483846 7406 raft_consensus.cc:1261] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [term 1 FOLLOWER]: Refusing update from remote peer 8cc9269e0ffd4c7b80d52a71e1490726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:41.483872 7542 raft_consensus.cc:1261] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [term 1 FOLLOWER]: Refusing update from remote peer 8cc9269e0ffd4c7b80d52a71e1490726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:41.484117 7823 consensus_queue.cc:1035] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6958b89a34dd480383a6e9cb5c83734c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:41.484270 7829 consensus_queue.cc:1035] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "096cc339e8854015a453c98d001e6707" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:41.485177 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.485032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:54599)
0927 21:12:41.485108 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.485142 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 21:12:41.485378 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.485257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:54321)
0927 21:12:41.485310 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.485347 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:12:41.486797 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.486611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:57151)
0927 21:12:41.486733 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.486764 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 21:12:41.487195 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.487107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:38609)
0927 21:12:41.487143 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.487167 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 21:12:41.488703 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.488580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:40519)
0927 21:12:41.488643 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.488670 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 21:12:41.488842 7085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:41.488935 7085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:41.490120 7085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87f53d45be014587b6578e1bfcb8dafe: Sending DeleteTablet for 3 replicas of tablet ec7e384b588f4c678bfdad43d3ac84b8
I20260927 21:12:41.490594 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7e384b588f4c678bfdad43d3ac84b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:41 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39830
I20260927 21:12:41.490677 7386 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7e384b588f4c678bfdad43d3ac84b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:41 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40046
I20260927 21:12:41.490818 7842 tablet_replica.cc:330] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: stopping tablet replica
I20260927 21:12:41.490857 7843 tablet_replica.cc:330] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c: stopping tablet replica
I20260927 21:12:41.490875 7842 raft_consensus.cc:2229] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:41.490907 7843 raft_consensus.cc:2229] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:41.490918 7842 raft_consensus.cc:2258] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:41.490950 7843 raft_consensus.cc:2258] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:41.490963 7658 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7e384b588f4c678bfdad43d3ac84b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:41 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:51124
I20260927 21:12:41.491237 7842 ts_tablet_manager.cc:1905] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:41.491338 7843 ts_tablet_manager.cc:1905] T ec7e384b588f4c678bfdad43d3ac84b8 P 6958b89a34dd480383a6e9cb5c83734c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:41.491474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 7323
I20260927 21:12:41.493010 7842 ts_tablet_manager.cc:1918] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:41.493076 7842 log.cc:1196] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/ec7e384b588f4c678bfdad43d3ac84b8
I20260927 21:12:41.493346 7842 ts_tablet_manager.cc:1939] T ec7e384b588f4c678bfdad43d3ac84b8 P 096cc339e8854015a453c98d001e6707: Deleting consensus metadata
I20260927 21:12:41.493724 7070 catalog_manager.cc:4841] TS 096cc339e8854015a453c98d001e6707 (127.0.98.2:34177): tablet ec7e384b588f4c678bfdad43d3ac84b8 (table test-table [id=e76d305623d4464195e350bcd55dc219]) successfully deleted
I20260927 21:12:41.496833 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:41.496700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35733 (local address 127.0.0.1:40052)
0927 21:12:41.496762 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:41.496791 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:12:41.496843 7844 tablet_replica.cc:330] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: stopping tablet replica
I20260927 21:12:41.496909 7844 raft_consensus.cc:2229] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:41.497013 7844 raft_consensus.cc:2258] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 21:12:41.497010 7070 catalog_manager.cc:4584] TS 6958b89a34dd480383a6e9cb5c83734c (127.0.98.1:35733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec7e384b588f4c678bfdad43d3ac84b8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35733: connect: Connection refused (error 111)
I20260927 21:12:41.497172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 7459
I20260927 21:12:41.497426 7844 ts_tablet_manager.cc:1905] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:41.499223 7844 ts_tablet_manager.cc:1918] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:41.499297 7844 log.cc:1196] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/ec7e384b588f4c678bfdad43d3ac84b8
I20260927 21:12:41.499519 7844 ts_tablet_manager.cc:1939] T ec7e384b588f4c678bfdad43d3ac84b8 P 8cc9269e0ffd4c7b80d52a71e1490726: Deleting consensus metadata
I20260927 21:12:41.500036 7069 catalog_manager.cc:4841] TS 8cc9269e0ffd4c7b80d52a71e1490726 (127.0.98.3:36143): tablet ec7e384b588f4c678bfdad43d3ac84b8 (table test-table [id=e76d305623d4464195e350bcd55dc219]) successfully deleted
I20260927 21:12:41.503628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 7595
I20260927 21:12:41.508283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 6951
I20260927 21:12:41.513379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 7139
2026-09-27T21:12:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4882 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:41 dist-test-slave-r9x1 krb5kdc[7847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:41 dist-test-slave-r9x1 krb5kdc[7847](info): set up 2 sockets
Sep 27 21:12:41 dist-test-slave-r9x1 krb5kdc[7847](info): commencing operation
krb5kdc: starting...
W20260927 21:12:42.477533 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:42 dist-test-slave-r9x1 krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 21:12:42.597793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46835
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:42.677855 7860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:42.678088 7860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:42.678112 7860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:42.679382 7860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:42.679438 7860 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:12:42.679450 7860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:42.679462 7860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:42.679473 7860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:42.679484 7860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:42.681085 7860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46835
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:42.681376 7860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:42.681618 7860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:42.686815 7868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:42.686842 7866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:42.686834 7865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:42.686872 7860 server_base.cc:1034] running on GCE node
I20260927 21:12:42.687320 7860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:42.687635 7860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:42.688783 7860 hybrid_clock.cc:648] HybridClock initialized: now 1790543562688775 us; error 33 us; skew 500 ppm
Sep 27 21:12:42 dist-test-slave-r9x1 krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543562, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:42.691551 7860 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 21:12:42.692028 7860 webserver.cc:460] Webserver started at http://127.0.98.62:41797/ using document root <none> and password file <none>
I20260927 21:12:42.692274 7860 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:42.692322 7860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:42.692447 7860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:42.693764 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "687d2e97a8c64094bd6e87c3e299e326"
format_stamp: "Formatted at 2026-09-27 21:12:42 on dist-test-slave-r9x1"
server_key: "c61defcaa8d0fbd2b589f424f0029e34"
server_key_iv: "f4b644f7a5132cdf8a939edf64c05773"
server_key_version: "encryptionkey@0"
I20260927 21:12:42.694150 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "687d2e97a8c64094bd6e87c3e299e326"
format_stamp: "Formatted at 2026-09-27 21:12:42 on dist-test-slave-r9x1"
server_key: "c61defcaa8d0fbd2b589f424f0029e34"
server_key_iv: "f4b644f7a5132cdf8a939edf64c05773"
server_key_version: "encryptionkey@0"
I20260927 21:12:42.695568 7860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:42.696480 7875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:42.696662 7860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:42.696760 7860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "687d2e97a8c64094bd6e87c3e299e326"
format_stamp: "Formatted at 2026-09-27 21:12:42 on dist-test-slave-r9x1"
server_key: "c61defcaa8d0fbd2b589f424f0029e34"
server_key_iv: "f4b644f7a5132cdf8a939edf64c05773"
server_key_version: "encryptionkey@0"
I20260927 21:12:42.696843 7860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:42.707283 7860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:42.709357 7860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:42.709486 7860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:42.714016 7860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46835
I20260927 21:12:42.714063 7937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46835 every 8 connection(s)
I20260927 21:12:42.714395 7860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:42.715075 7938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:42.717552 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326: Bootstrap starting.
I20260927 21:12:42.718125 7938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:42.718392 7938 log.cc:824] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:42.719118 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326: No bootstrap required, opened a new log
I20260927 21:12:42.721179 7938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d2e97a8c64094bd6e87c3e299e326" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46835 } }
I20260927 21:12:42.721364 7938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:42.721410 7938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687d2e97a8c64094bd6e87c3e299e326, State: Initialized, Role: FOLLOWER
I20260927 21:12:42.721558 7938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d2e97a8c64094bd6e87c3e299e326" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46835 } }
I20260927 21:12:42.721619 7938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:42.721650 7938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:42.721697 7938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:42.722239 7938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d2e97a8c64094bd6e87c3e299e326" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46835 } }
I20260927 21:12:42.722333 7938 leader_election.cc:302] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 687d2e97a8c64094bd6e87c3e299e326; no voters:
I20260927 21:12:42.722508 7938 leader_election.cc:288] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:42.722553 7941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:42.722664 7941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [term 1 LEADER]: Becoming Leader. State: Replica: 687d2e97a8c64094bd6e87c3e299e326, State: Running, Role: LEADER
I20260927 21:12:42.722707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 7860
I20260927 21:12:42.722779 7938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:42.722806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:42.722851 7941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d2e97a8c64094bd6e87c3e299e326" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46835 } }
I20260927 21:12:42.723013 392 external_mini_cluster.cc:1428] Setting key ec37c5e082fad1f89fa3de0eda28b41e
I20260927 21:12:42.723271 7942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687d2e97a8c64094bd6e87c3e299e326. Latest consensus state: current_term: 1 leader_uuid: "687d2e97a8c64094bd6e87c3e299e326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d2e97a8c64094bd6e87c3e299e326" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46835 } } }
I20260927 21:12:42.723448 7942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:42.723845 7951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:42.724131 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "687d2e97a8c64094bd6e87c3e299e326" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687d2e97a8c64094bd6e87c3e299e326" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46835 } } }
I20260927 21:12:42.724211 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326 [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:42.724366 7951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:42.726056 7951 catalog_manager.cc:1347] Generated new cluster ID: 5a225fef70f441eaa94f95410154bf09
I20260927 21:12:42.726110 7951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:12:42 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:42.733470 7951 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:42.734176 7951 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:42.736241 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:42.723715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.0.98.62:46835)
0927 21:12:42.724524 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:42.724533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:42.724650 (+ 117us) server_negotiation.cc:407] Connection header received
0927 21:12:42.724951 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:42.724961 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:42.725036 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:42.725171 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:42.725910 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.727421 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.728564 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.729246 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.729445 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:42.732376 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:42.732381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:42.732381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:42.732397 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:42.734370 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:42.734840 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:42.734842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:42.734843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:42.734951 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:42.735148 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:42.735149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:42.735149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:42.735570 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:42.735734 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:42.735835 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 21:12:42.735993 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":760,"thread_start_us":86,"threads_started":1}
I20260927 21:12:42.744478 7951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326: Generated new TSK 0
I20260927 21:12:42.744808 7951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 21:12:42.777525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46835
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:42.860457 7964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:42.860692 7964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:42.860715 7964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:42.861949 7964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:42.861994 7964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:42.862015 7964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:12:42.862038 7964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:42.863544 7964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:42.863756 7964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:42.863942 7964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:42.868784 7969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:42.868853 7964 server_base.cc:1034] running on GCE node
W20260927 21:12:42.868860 7972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:42.868784 7970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:42.869194 7964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:42.869472 7964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:42.870623 7964 hybrid_clock.cc:648] HybridClock initialized: now 1790543562870611 us; error 31 us; skew 500 ppm
Sep 27 21:12:42 dist-test-slave-r9x1 krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543562, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:42.877948 7964 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 21:12:42.878388 7964 webserver.cc:460] Webserver started at http://127.0.98.1:44441/ using document root <none> and password file <none>
I20260927 21:12:42.878611 7964 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:42.878661 7964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:42.878790 7964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:42.879853 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad"
format_stamp: "Formatted at 2026-09-27 21:12:42 on dist-test-slave-r9x1"
server_key: "f0c932fd8a8a6d9d9ca115955c0c6c5d"
server_key_iv: "c17c844bcf5aa4107bbc365a93cfef86"
server_key_version: "encryptionkey@0"
I20260927 21:12:42.880188 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad"
format_stamp: "Formatted at 2026-09-27 21:12:42 on dist-test-slave-r9x1"
server_key: "f0c932fd8a8a6d9d9ca115955c0c6c5d"
server_key_iv: "c17c844bcf5aa4107bbc365a93cfef86"
server_key_version: "encryptionkey@0"
I20260927 21:12:42.881557 7964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:42.882323 7979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:42.882493 7964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:42.882586 7964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad"
format_stamp: "Formatted at 2026-09-27 21:12:42 on dist-test-slave-r9x1"
server_key: "f0c932fd8a8a6d9d9ca115955c0c6c5d"
server_key_iv: "c17c844bcf5aa4107bbc365a93cfef86"
server_key_version: "encryptionkey@0"
I20260927 21:12:42.882650 7964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:42.893491 7964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:42.895538 7964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:42.895694 7964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:42.896196 7964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:42.896242 7964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:42.896277 7964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:42.896296 7964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:12:42 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543562, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:42.903170 7964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34375
I20260927 21:12:42.903225 8094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34375 every 8 connection(s)
I20260927 21:12:42.903559 7964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:12:42.905808 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:42.896779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41285 (local address 127.0.98.62:46835)
0927 21:12:42.896833 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:42.896835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:42.897034 (+ 199us) server_negotiation.cc:407] Connection header received
0927 21:12:42.897631 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:42.897632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:42.897646 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:42.897697 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:42.898359 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.899042 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.900192 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.900439 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.900505 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:42.902850 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:42.902851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:42.902852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:42.902860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:42.904429 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:42.904844 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:42.904845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:42.904845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:42.904908 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:42.905160 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:42.905161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:42.905161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:42.905330 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:42.905392 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:42.905663 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 21:12:42.905743 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:12:42.905982 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:42.896751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46835 (local address 127.0.98.1:41285)
0927 21:12:42.896913 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:42.896951 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:42.897521 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:42.897770 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:42.897775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:42.898060 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:42.898263 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:42.898268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.899142 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.899144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:42.900112 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:42.900115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.900539 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.900540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:42.900675 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:42.901231 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:42.901242 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:42.902736 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:42.904513 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:42.904518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:42.904520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:42.904753 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:42.905009 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:42.905010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:42.905010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:42.905113 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:42.905388 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:42.905394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:42.905550 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 21:12:42.905644 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:12:42.905777 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260927 21:12:42.906481 8095 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46835
I20260927 21:12:42.906567 8095 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:42.906761 8095 heartbeater.cc:511] Master 127.0.98.62:46835 requested a full tablet report, sending...
I20260927 21:12:42.907415 7891 ts_manager.cc:194] Registered new tserver with Master: eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375)
I20260927 21:12:42.908118 7891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41285
I20260927 21:12:42.912340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 7964
I20260927 21:12:42.912464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:42.912640 392 external_mini_cluster.cc:1428] Setting key dae318d7a0a047b7b68b3fbf76264677
I20260927 21:12:42.912940 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:42.907566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59959 (local address 127.0.98.62:46835)
0927 21:12:42.907604 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:42.907606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:42.907721 (+ 115us) server_negotiation.cc:407] Connection header received
0927 21:12:42.907776 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:42.907777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:42.907788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:42.907833 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:42.908143 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.908747 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.909625 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.909821 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.909873 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:42.910789 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:42.910790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:42.910791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:42.910798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:42.911906 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:42.912231 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:42.912232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:42.912232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:42.912288 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:42.912443 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:42.912444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:42.912444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:42.912571 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:42.912630 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:42.912766 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 21:12:42.912844 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:12:42.912952 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:42.907518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46835 (local address 127.0.98.1:59959)
0927 21:12:42.907654 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:42.907664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:42.907722 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:42.907836 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:42.907837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:42.908014 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:42.908077 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:42.908079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.908836 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.908837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:42.909529 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:42.909532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:42.909877 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:42.909878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:42.909957 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:42.910391 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:42.910396 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:42.910707 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:42.911985 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:42.911986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:42.911987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:42.912171 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:42.912339 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:42.912340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:42.912340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:42.912395 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:42.912621 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:42.912622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:42.912682 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 21:12:42.912743 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:12:42.912844 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 21:12:42.940771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46835
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:43.021915 8104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:43.022145 8104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:43.022171 8104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:43.023427 8104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:43.023478 8104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:43.023525 8104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:12:43.023551 8104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:43.025152 8104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:43.025384 8104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:43.025583 8104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:43.030580 8109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:43.030591 8112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:43.030596 8110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:43.030699 8104 server_base.cc:1034] running on GCE node
I20260927 21:12:43.030993 8104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:43.031270 8104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:43.032425 8104 hybrid_clock.cc:648] HybridClock initialized: now 1790543563032409 us; error 29 us; skew 500 ppm
Sep 27 21:12:43 dist-test-slave-r9x1 krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543563, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:43.035076 8104 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 21:12:43.035554 8104 webserver.cc:460] Webserver started at http://127.0.98.2:35103/ using document root <none> and password file <none>
I20260927 21:12:43.035753 8104 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:43.035795 8104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:43.035913 8104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:43.036998 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "74d59d5b52ef4fd1940c789748b61757"
format_stamp: "Formatted at 2026-09-27 21:12:43 on dist-test-slave-r9x1"
server_key: "755a691fca46cc417ea354dee6837ea6"
server_key_iv: "d378f863a691f9f9309ef1789ee17528"
server_key_version: "encryptionkey@0"
I20260927 21:12:43.037371 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "74d59d5b52ef4fd1940c789748b61757"
format_stamp: "Formatted at 2026-09-27 21:12:43 on dist-test-slave-r9x1"
server_key: "755a691fca46cc417ea354dee6837ea6"
server_key_iv: "d378f863a691f9f9309ef1789ee17528"
server_key_version: "encryptionkey@0"
I20260927 21:12:43.038727 8104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:43.039616 8119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.039790 8104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:43.039881 8104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "74d59d5b52ef4fd1940c789748b61757"
format_stamp: "Formatted at 2026-09-27 21:12:43 on dist-test-slave-r9x1"
server_key: "755a691fca46cc417ea354dee6837ea6"
server_key_iv: "d378f863a691f9f9309ef1789ee17528"
server_key_version: "encryptionkey@0"
I20260927 21:12:43.039948 8104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:43.050575 8104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:43.052608 8104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:43.052750 8104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:43.053253 8104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:43.053326 8104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.053367 8104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:43.053392 8104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:12:43 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543563, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:43.060109 8104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38149
I20260927 21:12:43.060175 8234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38149 every 8 connection(s)
I20260927 21:12:43.060531 8104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:12:43.063264 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.053751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60129 (local address 127.0.98.62:46835)
0927 21:12:43.053819 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.053821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.054034 (+ 213us) server_negotiation.cc:407] Connection header received
0927 21:12:43.054729 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.054731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.054743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.054798 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:43.055686 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.056452 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.057791 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.058008 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.058066 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.060289 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:43.060291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:43.060291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:43.060301 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:43.061843 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.062288 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.062289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.062289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.062355 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.062657 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.062658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.062658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.062792 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:43.062857 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.063108 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.063197 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:12:43.063438 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.053725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46835 (local address 127.0.98.2:60129)
0927 21:12:43.053894 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.053943 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.054600 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.054896 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.054901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.055273 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:43.055586 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.055595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.056546 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.056547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.057699 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.057703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.058094 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.058095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.058222 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.058722 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:43.058731 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:43.060167 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:43.061939 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.061946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.061948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.062193 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.062469 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.062469 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.062470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.062565 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.062862 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:43.062867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:43.063000 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.063087 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.063223 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260927 21:12:43.063898 8235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46835
I20260927 21:12:43.063982 8235 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:43.064179 8235 heartbeater.cc:511] Master 127.0.98.62:46835 requested a full tablet report, sending...
I20260927 21:12:43.064574 7892 ts_manager.cc:194] Registered new tserver with Master: 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149)
I20260927 21:12:43.064996 7892 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60129
I20260927 21:12:43.065569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 8104
I20260927 21:12:43.065670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:43.065833 392 external_mini_cluster.cc:1428] Setting key 5f704335e06ce66b54897ef4cca9548c
I20260927 21:12:43.070420 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.064859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46835 (local address 127.0.98.2:42665)
0927 21:12:43.064986 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.064996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.065048 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.065199 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.065199 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.065445 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:43.065554 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.065557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.066368 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.066369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.066898 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.066900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.067234 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.067235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.067301 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.067737 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:43.067741 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:43.068025 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:43.069447 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.069448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.069449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.069622 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.069795 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.069795 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.069796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.069854 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.070098 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:43.070099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:43.070145 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.070206 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.070314 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":10,"thread_start_us":70,"threads_started":1}
I20260927 21:12:43.070410 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.064921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42665 (local address 127.0.98.62:46835)
0927 21:12:43.064959 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.064961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.065050 (+ 89us) server_negotiation.cc:407] Connection header received
0927 21:12:43.065094 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.065095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.065105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.065175 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:43.065637 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.066290 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.066970 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.067173 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.067225 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.068103 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:43.068104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:43.068104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:43.068111 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:43.069379 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.069689 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.069690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.069691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.069745 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.069901 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.069902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.069903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.070044 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:43.070100 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.070233 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.070311 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 21:12:43.092825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46835
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:43.177323 8244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:43.177613 8244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:43.177646 8244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:43.178996 8244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:43.179060 8244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:43.179081 8244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:12:43.179098 8244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:43.180755 8244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:43.181023 8244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:43.181257 8244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:43.186414 8250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:43.186407 8249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:43.186434 8244 server_base.cc:1034] running on GCE node
W20260927 21:12:43.186407 8252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:43.186851 8244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:43.187127 8244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:43.188278 8244 hybrid_clock.cc:648] HybridClock initialized: now 1790543563188261 us; error 37 us; skew 500 ppm
Sep 27 21:12:43 dist-test-slave-r9x1 krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543563, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:43.190853 8244 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 21:12:43.191318 8244 webserver.cc:460] Webserver started at http://127.0.98.3:44817/ using document root <none> and password file <none>
I20260927 21:12:43.191540 8244 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:43.191587 8244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:43.191713 8244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:43.192905 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "dfcb09f80a5e4253adee88457fe7ce88"
format_stamp: "Formatted at 2026-09-27 21:12:43 on dist-test-slave-r9x1"
server_key: "31af777bf4a94b0c817d56c0067955b5"
server_key_iv: "59795b2c2f0ccb48112fbec17ef82092"
server_key_version: "encryptionkey@0"
I20260927 21:12:43.193317 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dfcb09f80a5e4253adee88457fe7ce88"
format_stamp: "Formatted at 2026-09-27 21:12:43 on dist-test-slave-r9x1"
server_key: "31af777bf4a94b0c817d56c0067955b5"
server_key_iv: "59795b2c2f0ccb48112fbec17ef82092"
server_key_version: "encryptionkey@0"
I20260927 21:12:43.194674 8244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:43.195480 8259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.195781 8244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:12:43.195883 8244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "dfcb09f80a5e4253adee88457fe7ce88"
format_stamp: "Formatted at 2026-09-27 21:12:43 on dist-test-slave-r9x1"
server_key: "31af777bf4a94b0c817d56c0067955b5"
server_key_iv: "59795b2c2f0ccb48112fbec17ef82092"
server_key_version: "encryptionkey@0"
I20260927 21:12:43.195959 8244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:43.212077 8244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:43.214345 8244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:43.214473 8244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:43.215144 8244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:43.215197 8244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.215219 8244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:43.215230 8244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.222048 8244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46711
I20260927 21:12:43.222116 8374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46711 every 8 connection(s)
Sep 27 21:12:43 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543563, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:43.222476 8244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:43.225797 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.215627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52949 (local address 127.0.98.62:46835)
0927 21:12:43.215688 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.215691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.215880 (+ 189us) server_negotiation.cc:407] Connection header received
0927 21:12:43.216531 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.216534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.216548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.216625 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:43.217498 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.218345 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.220074 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.220374 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.220464 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.223033 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:43.223034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:43.223035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:43.223044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:43.224504 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.224884 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.224885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.224886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.224934 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.225126 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.225127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.225127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.225226 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:43.225288 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.225665 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.225730 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:12:43.226030 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.215580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46835 (local address 127.0.98.3:52949)
0927 21:12:43.215750 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.215792 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.216405 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.216699 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.216702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.217087 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:43.217408 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.217414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.218463 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.218465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.219961 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.219969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.220464 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.220465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.220571 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.221143 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:43.221156 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:43.222922 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:43.224595 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.224602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.224604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.224829 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.224982 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.224983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.224983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.225086 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.225355 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:43.225362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:43.225543 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.225644 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.225782 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260927 21:12:43.226575 8375 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46835
I20260927 21:12:43.226838 8375 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:43.227092 8375 heartbeater.cc:511] Master 127.0.98.62:46835 requested a full tablet report, sending...
I20260927 21:12:43.227520 7892 ts_manager.cc:194] Registered new tserver with Master: dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711)
I20260927 21:12:43.227926 7892 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52949
I20260927 21:12:43.228288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 8244
I20260927 21:12:43.228390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:43.228574 392 external_mini_cluster.cc:1428] Setting key 1b855d51de836126ab577cea2c537f9f
I20260927 21:12:43.229421 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:12:43.233989 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.227577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46835 (local address 127.0.98.3:57001)
0927 21:12:43.227728 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.227738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.227791 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.227918 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.227919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.228121 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:43.228190 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.228192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.228937 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.228938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.229609 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.229611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.230035 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.230036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.230095 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.230550 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:43.230555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:43.230851 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:43.232381 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.232382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.232383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.232610 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.233015 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.233015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.233016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.233096 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.233616 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:43.233617 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:43.233679 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.233762 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.233883 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260927 21:12:43.233976 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.227630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57001 (local address 127.0.98.62:46835)
0927 21:12:43.227674 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.227675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.227794 (+ 119us) server_negotiation.cc:407] Connection header received
0927 21:12:43.227854 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.227855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.227864 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.227905 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:43.228253 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.228829 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.229682 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.229967 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.230038 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.230938 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:43.230939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:43.230939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:43.230947 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:43.232288 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.232771 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.232773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.232773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.232891 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.233178 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.233179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.233179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.233525 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:43.233620 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.233786 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.233879 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 21:12:43 dist-test-slave-r9x1 krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:12:43 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:43 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:12:43.250823 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:43.245391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.0.98.62:46835)
0927 21:12:43.245441 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.245443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.245452 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:43.245481 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.245481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.245489 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.245553 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:43.246104 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.246613 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.247368 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.247559 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.247620 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.250737 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50854: BlockingRecv error: recv got EOF from 127.0.0.1:50854 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 21:12:43 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:43.258562 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.251617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.0.98.62:46835)
0927 21:12:43.251662 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.251664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.251751 (+ 87us) server_negotiation.cc:407] Connection header received
0927 21:12:43.251819 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.251820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.251829 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.251876 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:43.252332 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.252812 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.253526 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.253710 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.253760 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.256337 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:43.256338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:43.256339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:43.256347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:43.257549 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.257879 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.257880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.257881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.257928 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.258091 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.258092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.258092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.258205 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:43.258291 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.258387 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.258465 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:12:43.260007 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:43.261658 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:43.268517 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.264471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.0.98.1:34375)
0927 21:12:43.264653 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.264658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.264746 (+ 88us) server_negotiation.cc:407] Connection header received
0927 21:12:43.264822 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.264824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.264839 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.264961 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:43.265122 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.266294 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.267537 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.268093 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.268159 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.268246 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.268364 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.268408 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260927 21:12:43.268500 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.264149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38149 (local address 127.0.0.1:45464)
0927 21:12:43.264450 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.264459 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.264496 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.265027 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.265028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.265030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:43.265133 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.265135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.266422 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.266423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.267313 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.267315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.268256 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.268257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.268322 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.268326 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.268355 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.268421 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":34,"thread_start_us":69,"threads_started":1}
I20260927 21:12:43.268661 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.264248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.0.98.2:38149)
0927 21:12:43.264675 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.264680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.264693 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:12:43.264908 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.264911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.264930 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.265213 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:43.265333 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.266364 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.267403 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.268167 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.268399 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.268486 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.268509 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.268563 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":98,"threads_started":1}
I20260927 21:12:43.268695 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.264475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.0.98.3:46711)
0927 21:12:43.264716 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.264719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.264728 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:43.264771 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.264773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.264788 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.264863 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:43.265095 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.265948 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.267383 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.268154 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.268250 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.268330 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.268561 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.268607 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260927 21:12:43.268790 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.264357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34375 (local address 127.0.0.1:48434)
0927 21:12:43.264661 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.264673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.264743 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.265017 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.265018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.265019 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:43.265048 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.265049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.266386 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.266387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.267461 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.267462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.268175 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.268176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.268260 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.268275 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.268345 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.268736 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":57,"threads_started":1}
I20260927 21:12:43.268965 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.264149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46711 (local address 127.0.0.1:51354)
0927 21:12:43.264277 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.264323 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.264396 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.264939 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.264942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.264944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:43.265017 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.265019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.266029 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.266030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.267289 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.267292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.268359 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.268360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.268428 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.268431 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.268473 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.268519 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260927 21:12:43.269553 8169 tablet_service.cc:1467] Processing CreateTablet for tablet c2b729d4cbbf4cab85950ec4f05c17b9 (DEFAULT_TABLE table=test-table [id=640061a0aa2c437e9b3ebb10d400f075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:43.269589 8029 tablet_service.cc:1467] Processing CreateTablet for tablet c2b729d4cbbf4cab85950ec4f05c17b9 (DEFAULT_TABLE table=test-table [id=640061a0aa2c437e9b3ebb10d400f075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:43.269634 8309 tablet_service.cc:1467] Processing CreateTablet for tablet c2b729d4cbbf4cab85950ec4f05c17b9 (DEFAULT_TABLE table=test-table [id=640061a0aa2c437e9b3ebb10d400f075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:43.269907 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b729d4cbbf4cab85950ec4f05c17b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:43.269925 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b729d4cbbf4cab85950ec4f05c17b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:43.269925 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b729d4cbbf4cab85950ec4f05c17b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:43.272698 8403 tablet_bootstrap.cc:492] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: Bootstrap starting.
I20260927 21:12:43.273393 8403 tablet_bootstrap.cc:654] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:43.273726 8405 tablet_bootstrap.cc:492] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Bootstrap starting.
I20260927 21:12:43.273729 8403 log.cc:824] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:43.274405 8405 tablet_bootstrap.cc:654] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:43.274483 8403 tablet_bootstrap.cc:492] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: No bootstrap required, opened a new log
I20260927 21:12:43.274569 8403 ts_tablet_manager.cc:1397] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:12:43.274735 8405 log.cc:824] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:43.275087 8404 tablet_bootstrap.cc:492] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: Bootstrap starting.
I20260927 21:12:43.275408 8405 tablet_bootstrap.cc:492] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: No bootstrap required, opened a new log
I20260927 21:12:43.275470 8405 ts_tablet_manager.cc:1397] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:43.275805 8404 tablet_bootstrap.cc:654] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:43.276139 8404 log.cc:824] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:43.276697 8403 raft_consensus.cc:348] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.276834 8403 raft_consensus.cc:374] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:43.276855 8403 raft_consensus.cc:729] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcb09f80a5e4253adee88457fe7ce88, State: Initialized, Role: FOLLOWER
I20260927 21:12:43.276958 8403 consensus_queue.cc:260] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.277120 8405 raft_consensus.cc:348] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.277226 8375 heartbeater.cc:503] Master 127.0.98.62:46835 was elected leader, sending a full tablet report...
I20260927 21:12:43.277244 8405 raft_consensus.cc:374] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:43.277303 8405 raft_consensus.cc:729] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0c5c7e4f2d4c149844a6b4957cf6ad, State: Initialized, Role: FOLLOWER
I20260927 21:12:43.277436 8405 consensus_queue.cc:260] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.277523 8403 ts_tablet_manager.cc:1428] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:12:43.277765 8095 heartbeater.cc:503] Master 127.0.98.62:46835 was elected leader, sending a full tablet report...
I20260927 21:12:43.277839 8404 tablet_bootstrap.cc:492] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: No bootstrap required, opened a new log
I20260927 21:12:43.277910 8404 ts_tablet_manager.cc:1397] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:12:43.277755 8405 ts_tablet_manager.cc:1428] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:12:43.279704 8404 raft_consensus.cc:348] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.279838 8404 raft_consensus.cc:374] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:43.279858 8404 raft_consensus.cc:729] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d59d5b52ef4fd1940c789748b61757, State: Initialized, Role: FOLLOWER
I20260927 21:12:43.279968 8404 consensus_queue.cc:260] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.280211 8404 ts_tablet_manager.cc:1428] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:12:43.280248 8235 heartbeater.cc:503] Master 127.0.98.62:46835 was elected leader, sending a full tablet report...
W20260927 21:12:43.311237 8236 tablet.cc:2369] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:12:43.404408 8096 tablet.cc:2369] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:43.435878 8409 raft_consensus.cc:482] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:43.436069 8410 raft_consensus.cc:482] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:43.436069 8409 raft_consensus.cc:504] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.436178 8410 raft_consensus.cc:504] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.436479 8409 leader_election.cc:288] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149), eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375)
I20260927 21:12:43.436503 8410 leader_election.cc:288] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711), 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149)
I20260927 21:12:43.440532 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.436685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38149 (local address 127.0.98.3:53925)
0927 21:12:43.436907 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.436916 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.436956 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.437216 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.437217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.437219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:43.437299 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.437302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.438227 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.438228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.439486 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.439490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.440283 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.440284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.440357 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.440360 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.440401 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.440439 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":33}
I20260927 21:12:43.440631 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.437131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42113 (local address 127.0.98.1:34375)
0927 21:12:43.437170 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.437172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.437178 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:12:43.437191 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.437192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.437200 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.437252 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:43.437456 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.438189 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.439593 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.440122 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.440187 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.440198 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.440471 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.440516 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:12:43.440650 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.436776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34375 (local address 127.0.98.3:42113)
0927 21:12:43.436925 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.436935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.436982 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.437308 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.437308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.437320 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:43.437383 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.437385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.438281 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.438282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.439489 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.439492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.440231 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.440233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.440318 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.440321 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.440364 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.440408 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260927 21:12:43.440650 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.436784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41811 (local address 127.0.98.2:38149)
0927 21:12:43.437214 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.437216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.437222 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:12:43.437240 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.437240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.437256 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.437332 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:43.437512 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.438261 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.439623 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.440232 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.440285 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.440296 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.440538 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.440579 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":404,"thread_start_us":77,"threads_started":1}
I20260927 21:12:43.440739 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.436870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40155 (local address 127.0.98.3:46711)
0927 21:12:43.436925 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.436927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.436969 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:12:43.436992 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.436993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.437013 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.437060 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:43.437391 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.438133 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.439596 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.440122 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.440168 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.440180 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.440389 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.440436 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:12:43.440831 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.436707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38149 (local address 127.0.98.1:41811)
0927 21:12:43.437067 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.437076 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.437117 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.437374 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.437375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.437377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:43.437442 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.437443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.438340 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.438341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.439493 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.439495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.440326 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.440327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.440397 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.440400 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.440440 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.440484 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260927 21:12:43.440845 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.436675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46711 (local address 127.0.98.1:40155)
0927 21:12:43.436851 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.436862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.436921 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.437090 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.437091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.437094 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:43.437314 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.437316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.438229 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.438230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.439489 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.439492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.440239 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.440240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.440312 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.440316 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.440455 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.440504 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260927 21:12:43.441138 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.436765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53925 (local address 127.0.98.2:38149)
0927 21:12:43.436824 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.436827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.437082 (+ 255us) server_negotiation.cc:407] Connection header received
0927 21:12:43.437110 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.437111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.437123 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.437172 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:43.437372 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.438136 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.439598 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.440195 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.440247 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.440256 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.441030 (+ 774us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.441077 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":33}
I20260927 21:12:43.441160 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b729d4cbbf4cab85950ec4f05c17b9" candidate_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d59d5b52ef4fd1940c789748b61757" is_pre_election: true
I20260927 21:12:43.441387 8189 raft_consensus.cc:2454] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb0c5c7e4f2d4c149844a6b4957cf6ad in term 0.
I20260927 21:12:43.441460 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b729d4cbbf4cab85950ec4f05c17b9" candidate_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb09f80a5e4253adee88457fe7ce88" is_pre_election: true
I20260927 21:12:43.441558 8329 raft_consensus.cc:2454] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb0c5c7e4f2d4c149844a6b4957cf6ad in term 0.
I20260927 21:12:43.441532 8188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b729d4cbbf4cab85950ec4f05c17b9" candidate_uuid: "dfcb09f80a5e4253adee88457fe7ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d59d5b52ef4fd1940c789748b61757" is_pre_election: true
I20260927 21:12:43.441627 8188 raft_consensus.cc:2454] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfcb09f80a5e4253adee88457fe7ce88 in term 0.
I20260927 21:12:43.441620 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b729d4cbbf4cab85950ec4f05c17b9" candidate_uuid: "dfcb09f80a5e4253adee88457fe7ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" is_pre_election: true
I20260927 21:12:43.441713 8049 raft_consensus.cc:2454] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfcb09f80a5e4253adee88457fe7ce88 in term 0.
I20260927 21:12:43.441745 7983 leader_election.cc:302] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfcb09f80a5e4253adee88457fe7ce88, eb0c5c7e4f2d4c149844a6b4957cf6ad; no voters:
I20260927 21:12:43.441851 8260 leader_election.cc:302] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d59d5b52ef4fd1940c789748b61757, dfcb09f80a5e4253adee88457fe7ce88; no voters:
I20260927 21:12:43.441954 8410 raft_consensus.cc:2781] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:43.441979 8409 raft_consensus.cc:2781] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:43.442027 8410 raft_consensus.cc:482] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:43.442036 8409 raft_consensus.cc:482] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:43.442055 8410 raft_consensus.cc:3037] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:43.442060 8409 raft_consensus.cc:3037] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:43.442790 8409 raft_consensus.cc:504] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.442807 8410 raft_consensus.cc:504] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.442914 8409 leader_election.cc:288] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [CANDIDATE]: Term 1 election: Requested vote from peers 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149), eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375)
I20260927 21:12:43.442930 8410 leader_election.cc:288] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [CANDIDATE]: Term 1 election: Requested vote from peers dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711), 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149)
I20260927 21:12:43.443099 8188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b729d4cbbf4cab85950ec4f05c17b9" candidate_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d59d5b52ef4fd1940c789748b61757"
I20260927 21:12:43.443132 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b729d4cbbf4cab85950ec4f05c17b9" candidate_uuid: "dfcb09f80a5e4253adee88457fe7ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d59d5b52ef4fd1940c789748b61757"
I20260927 21:12:43.443132 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b729d4cbbf4cab85950ec4f05c17b9" candidate_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb09f80a5e4253adee88457fe7ce88"
I20260927 21:12:43.443212 8329 raft_consensus.cc:2379] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb0c5c7e4f2d4c149844a6b4957cf6ad in current term 1: Already voted for candidate dfcb09f80a5e4253adee88457fe7ce88 in this term.
I20260927 21:12:43.443181 8188 raft_consensus.cc:3037] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:43.443930 8188 raft_consensus.cc:2454] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0c5c7e4f2d4c149844a6b4957cf6ad in term 1.
I20260927 21:12:43.444164 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b729d4cbbf4cab85950ec4f05c17b9" candidate_uuid: "dfcb09f80a5e4253adee88457fe7ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad"
I20260927 21:12:43.444171 7980 leader_election.cc:302] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74d59d5b52ef4fd1940c789748b61757, eb0c5c7e4f2d4c149844a6b4957cf6ad; no voters: dfcb09f80a5e4253adee88457fe7ce88
I20260927 21:12:43.444275 8049 raft_consensus.cc:2379] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dfcb09f80a5e4253adee88457fe7ce88 in current term 1: Already voted for candidate eb0c5c7e4f2d4c149844a6b4957cf6ad in this term.
I20260927 21:12:43.444347 8410 raft_consensus.cc:2781] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:43.444463 8410 raft_consensus.cc:686] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 LEADER]: Becoming Leader. State: Replica: eb0c5c7e4f2d4c149844a6b4957cf6ad, State: Running, Role: LEADER
I20260927 21:12:43.444552 8410 consensus_queue.cc:237] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } }
I20260927 21:12:43.444993 8260 leader_election.cc:302] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfcb09f80a5e4253adee88457fe7ce88; no voters: 74d59d5b52ef4fd1940c789748b61757, eb0c5c7e4f2d4c149844a6b4957cf6ad
I20260927 21:12:43.445111 8409 raft_consensus.cc:2726] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 21:12:43.445475 7891 catalog_manager.cc:5495] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad reported cstate change: term changed from 0 to 1, leader changed from <none> to eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1). New cstate: current_term: 1 leader_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } health_report { overall_health: HEALTHY } } }
I20260927 21:12:43.454777 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.452462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.0.98.1:34375)
0927 21:12:43.452521 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.452523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.452532 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:43.452573 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.452574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.452591 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.452652 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:43.452778 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.453312 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.454053 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.454239 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.454295 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.454458 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:43.454567 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:43.454648 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.454670 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.454693 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:12:43.456653 8329 raft_consensus.cc:1261] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Refusing update from remote peer eb0c5c7e4f2d4c149844a6b4957cf6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:43.456653 8188 raft_consensus.cc:1261] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Refusing update from remote peer eb0c5c7e4f2d4c149844a6b4957cf6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:43.456939 8416 consensus_queue.cc:1035] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:43.457078 8410 consensus_queue.cc:1035] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:43.460143 8420 mvcc.cc:205] Tried to move back new op lower bound from 7334066435915665408 to 7334066435869507584. Current Snapshot: MvccSnapshot[applied={T|T < 7334066435915665408 or (T in {7334066435915665408})}]
I20260927 21:12:43.462051 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:43.462160 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:43.462870 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326: Sending DeleteTablet for 3 replicas of tablet c2b729d4cbbf4cab85950ec4f05c17b9
I20260927 21:12:43.463204 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b729d4cbbf4cab85950ec4f05c17b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51354
I20260927 21:12:43.463302 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b729d4cbbf4cab85950ec4f05c17b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45464
I20260927 21:12:43.463304 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b729d4cbbf4cab85950ec4f05c17b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48434
I20260927 21:12:43.463541 8430 tablet_replica.cc:330] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: stopping tablet replica
I20260927 21:12:43.463650 8430 raft_consensus.cc:2229] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:43.463617 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:43.463752 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:43.463773 8430 raft_consensus.cc:2258] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:43.463935 8431 tablet_replica.cc:330] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: stopping tablet replica
I20260927 21:12:43.464036 8431 raft_consensus.cc:2229] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:43.464041 8429 tablet_replica.cc:330] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: stopping tablet replica
I20260927 21:12:43.464118 8431 raft_consensus.cc:2258] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:43.464139 8429 raft_consensus.cc:2229] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:43.464247 8429 raft_consensus.cc:2258] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:43.464491 8430 ts_tablet_manager.cc:1905] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:43.464681 8429 ts_tablet_manager.cc:1905] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:43.465607 8169 tablet_service.cc:1467] Processing CreateTablet for tablet bc1e06d199e84275adf63e8bddc56a52 (DEFAULT_TABLE table=test-table [id=0dde72959e6248e8a0669af22f1669c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:43.465687 8029 tablet_service.cc:1467] Processing CreateTablet for tablet bc1e06d199e84275adf63e8bddc56a52 (DEFAULT_TABLE table=test-table [id=0dde72959e6248e8a0669af22f1669c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:43.465714 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1e06d199e84275adf63e8bddc56a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:43.465713 8309 tablet_service.cc:1467] Processing CreateTablet for tablet bc1e06d199e84275adf63e8bddc56a52 (DEFAULT_TABLE table=test-table [id=0dde72959e6248e8a0669af22f1669c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:43.465780 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1e06d199e84275adf63e8bddc56a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:43.466524 8430 ts_tablet_manager.cc:1918] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:43.466593 8430 log.cc:1196] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/c2b729d4cbbf4cab85950ec4f05c17b9
I20260927 21:12:43.466831 8430 ts_tablet_manager.cc:1939] T c2b729d4cbbf4cab85950ec4f05c17b9 P 74d59d5b52ef4fd1940c789748b61757: Deleting consensus metadata
I20260927 21:12:43.466986 8404 tablet_bootstrap.cc:492] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: Bootstrap starting.
I20260927 21:12:43.467017 8405 tablet_bootstrap.cc:492] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Bootstrap starting.
I20260927 21:12:43.467389 8404 tablet_bootstrap.cc:654] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:43.467433 8405 tablet_bootstrap.cc:654] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:43.467849 7876 catalog_manager.cc:4841] TS 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149): tablet c2b729d4cbbf4cab85950ec4f05c17b9 (table test-table [id=640061a0aa2c437e9b3ebb10d400f075]) successfully deleted
I20260927 21:12:43.467963 8404 tablet_bootstrap.cc:492] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: No bootstrap required, opened a new log
I20260927 21:12:43.468026 8404 ts_tablet_manager.cc:1397] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:43.468176 8404 raft_consensus.cc:348] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.468253 8404 raft_consensus.cc:374] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:43.468281 8404 raft_consensus.cc:729] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d59d5b52ef4fd1940c789748b61757, State: Initialized, Role: FOLLOWER
I20260927 21:12:43.468322 8404 consensus_queue.cc:260] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.468407 8404 ts_tablet_manager.cc:1428] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.468436 8405 tablet_bootstrap.cc:492] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: No bootstrap required, opened a new log
I20260927 21:12:43.468492 8405 ts_tablet_manager.cc:1397] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:43.468518 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1e06d199e84275adf63e8bddc56a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:43.468595 8431 ts_tablet_manager.cc:1905] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:43.468649 8405 raft_consensus.cc:348] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.468729 8405 raft_consensus.cc:374] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:43.468753 8405 raft_consensus.cc:729] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0c5c7e4f2d4c149844a6b4957cf6ad, State: Initialized, Role: FOLLOWER
I20260927 21:12:43.468793 8405 consensus_queue.cc:260] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.468875 8405 ts_tablet_manager.cc:1428] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.470005 8429 ts_tablet_manager.cc:1918] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:43.470091 8429 log.cc:1196] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/c2b729d4cbbf4cab85950ec4f05c17b9
I20260927 21:12:43.470177 8403 tablet_bootstrap.cc:492] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: Bootstrap starting.
I20260927 21:12:43.470355 8429 ts_tablet_manager.cc:1939] T c2b729d4cbbf4cab85950ec4f05c17b9 P dfcb09f80a5e4253adee88457fe7ce88: Deleting consensus metadata
I20260927 21:12:43.470594 8403 tablet_bootstrap.cc:654] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:43.470661 8431 ts_tablet_manager.cc:1918] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:43.470731 8431 log.cc:1196] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/c2b729d4cbbf4cab85950ec4f05c17b9
I20260927 21:12:43.470968 8431 ts_tablet_manager.cc:1939] T c2b729d4cbbf4cab85950ec4f05c17b9 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Deleting consensus metadata
I20260927 21:12:43.471287 8403 tablet_bootstrap.cc:492] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: No bootstrap required, opened a new log
I20260927 21:12:43.471331 8403 ts_tablet_manager.cc:1397] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:43.471475 8403 raft_consensus.cc:348] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.471560 8403 raft_consensus.cc:374] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:43.471580 8403 raft_consensus.cc:729] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcb09f80a5e4253adee88457fe7ce88, State: Initialized, Role: FOLLOWER
I20260927 21:12:43.471613 8403 consensus_queue.cc:260] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.471695 8403 ts_tablet_manager.cc:1428] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.471674 7879 catalog_manager.cc:4841] TS dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711): tablet c2b729d4cbbf4cab85950ec4f05c17b9 (table test-table [id=640061a0aa2c437e9b3ebb10d400f075]) successfully deleted
I20260927 21:12:43.471884 7876 catalog_manager.cc:4841] TS eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375): tablet c2b729d4cbbf4cab85950ec4f05c17b9 (table test-table [id=640061a0aa2c437e9b3ebb10d400f075]) successfully deleted
W20260927 21:12:43.473006 8376 tablet.cc:2369] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:12:43.623633 8411 raft_consensus.cc:482] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:43.623806 8411 raft_consensus.cc:504] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.624243 8411 leader_election.cc:288] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375), dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711)
I20260927 21:12:43.627347 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.624489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58951 (local address 127.0.98.3:46711)
0927 21:12:43.624536 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.624538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.624659 (+ 121us) server_negotiation.cc:407] Connection header received
0927 21:12:43.624720 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.624721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.624730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.624773 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:43.624915 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.625427 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.626586 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.627014 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.627084 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.627095 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.627235 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.627273 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:12:43.627370 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.624431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46711 (local address 127.0.98.2:58951)
0927 21:12:43.624587 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.624597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.624660 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.624784 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.624785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.624788 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:43.624858 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.624860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.625518 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.625519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.626487 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.626490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.627089 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.627090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.627171 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.627175 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.627222 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.627270 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":32,"thread_start_us":59,"threads_started":1}
I20260927 21:12:43.627365 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.624487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60795 (local address 127.0.98.1:34375)
0927 21:12:43.624542 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.624544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.624674 (+ 130us) server_negotiation.cc:407] Connection header received
0927 21:12:43.624737 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.624738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.624753 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.624802 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:12:43.624933 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.625450 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.626580 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.626996 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.627105 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.627117 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.627246 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.627300 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:12:43.627516 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.624431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34375 (local address 127.0.98.2:60795)
0927 21:12:43.624589 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.624599 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.624685 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.624812 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.624813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.624815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:12:43.624876 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.624878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.625543 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.625544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.626488 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.626491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.627113 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.627114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.627185 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.627189 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.627235 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.627287 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":68,"thread_start_us":66,"threads_started":1}
I20260927 21:12:43.627729 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1e06d199e84275adf63e8bddc56a52" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb09f80a5e4253adee88457fe7ce88" is_pre_election: true
I20260927 21:12:43.627758 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1e06d199e84275adf63e8bddc56a52" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" is_pre_election: true
I20260927 21:12:43.627827 8329 raft_consensus.cc:2454] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 0.
I20260927 21:12:43.627847 8049 raft_consensus.cc:2454] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 0.
I20260927 21:12:43.628007 8123 leader_election.cc:302] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d59d5b52ef4fd1940c789748b61757, dfcb09f80a5e4253adee88457fe7ce88; no voters:
I20260927 21:12:43.628185 8411 raft_consensus.cc:2781] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:43.628224 8411 raft_consensus.cc:482] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:43.628243 8411 raft_consensus.cc:3037] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:43.628875 8411 raft_consensus.cc:504] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.629019 8411 leader_election.cc:288] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 election: Requested vote from peers eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375), dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711)
I20260927 21:12:43.629184 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1e06d199e84275adf63e8bddc56a52" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb09f80a5e4253adee88457fe7ce88"
I20260927 21:12:43.629197 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1e06d199e84275adf63e8bddc56a52" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad"
I20260927 21:12:43.629302 8329 raft_consensus.cc:3037] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:43.629302 8049 raft_consensus.cc:3037] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:43.630101 8049 raft_consensus.cc:2454] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 1.
I20260927 21:12:43.630101 8329 raft_consensus.cc:2454] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 1.
I20260927 21:12:43.630337 8120 leader_election.cc:302] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d59d5b52ef4fd1940c789748b61757, eb0c5c7e4f2d4c149844a6b4957cf6ad; no voters:
I20260927 21:12:43.630426 8411 raft_consensus.cc:2781] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:43.630563 8411 raft_consensus.cc:686] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 1 LEADER]: Becoming Leader. State: Replica: 74d59d5b52ef4fd1940c789748b61757, State: Running, Role: LEADER
I20260927 21:12:43.630626 8411 consensus_queue.cc:237] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.631506 7891 catalog_manager.cc:5495] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74d59d5b52ef4fd1940c789748b61757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:12:43 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543562, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:43.684515 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.678124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.0.98.62:46835)
0927 21:12:43.678188 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:43.678190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:43.678299 (+ 109us) server_negotiation.cc:407] Connection header received
0927 21:12:43.678329 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:43.678330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:43.678340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:43.678397 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:43.678798 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.679302 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.680054 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.680239 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.680297 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.682283 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:43.682285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:43.682285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:43.682296 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:43.683423 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.683810 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.683811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.683811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.683870 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:43.684033 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:43.684033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:43.684034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:43.684134 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:43.684188 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:43.684319 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:12:43.684398 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:12:43.684584 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:43.678060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46835 (local address 127.0.0.1:50886)
0927 21:12:43.678211 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:43.678221 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:43.678286 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:43.678409 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:43.678409 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:43.678691 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:43.678739 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.678741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.679392 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.679393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.679941 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:43.679943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:43.680310 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:43.680311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:43.680352 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:43.680829 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:43.680833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:43.682191 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:43.683511 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.683518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.683521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.683723 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.683925 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:43.683926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:43.683926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:43.683983 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:43.684187 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:43.684196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:43.684238 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:12:43.684291 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:12:43.684465 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":72,"thread_start_us":78,"threads_started":1}
I20260927 21:12:43.686609 7891 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50886:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:12:43.686821 7891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:43.688169 8029 tablet_service.cc:1467] Processing CreateTablet for tablet b2b7a40aa5fb4fc8b1a05cca713e9870 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c23e80d4f4064f8a8568a9f811f9140e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:12:43.688206 8169 tablet_service.cc:1467] Processing CreateTablet for tablet b2b7a40aa5fb4fc8b1a05cca713e9870 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c23e80d4f4064f8a8568a9f811f9140e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:12:43.688334 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b7a40aa5fb4fc8b1a05cca713e9870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:43.688334 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b7a40aa5fb4fc8b1a05cca713e9870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:43.689451 8309 tablet_service.cc:1467] Processing CreateTablet for tablet b2b7a40aa5fb4fc8b1a05cca713e9870 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c23e80d4f4064f8a8568a9f811f9140e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:12:43.689604 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b7a40aa5fb4fc8b1a05cca713e9870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:43.689872 8404 tablet_bootstrap.cc:492] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: Bootstrap starting.
I20260927 21:12:43.689875 8405 tablet_bootstrap.cc:492] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Bootstrap starting.
I20260927 21:12:43.690215 8405 tablet_bootstrap.cc:654] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:43.690234 8404 tablet_bootstrap.cc:654] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:43.690752 8405 tablet_bootstrap.cc:492] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: No bootstrap required, opened a new log
I20260927 21:12:43.690809 8405 ts_tablet_manager.cc:1397] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:43.690930 8405 raft_consensus.cc:348] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.691009 8405 raft_consensus.cc:374] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:43.691028 8405 raft_consensus.cc:729] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0c5c7e4f2d4c149844a6b4957cf6ad, State: Initialized, Role: FOLLOWER
I20260927 21:12:43.691077 8405 consensus_queue.cc:260] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.691133 8403 tablet_bootstrap.cc:492] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: Bootstrap starting.
I20260927 21:12:43.691167 8405 ts_tablet_manager.cc:1428] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.691490 8404 tablet_bootstrap.cc:492] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: No bootstrap required, opened a new log
I20260927 21:12:43.691515 8403 tablet_bootstrap.cc:654] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:43.691542 8404 ts_tablet_manager.cc:1397] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:12:43.691702 8404 raft_consensus.cc:348] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.691766 8404 raft_consensus.cc:374] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:43.691779 8404 raft_consensus.cc:729] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d59d5b52ef4fd1940c789748b61757, State: Initialized, Role: FOLLOWER
I20260927 21:12:43.691813 8404 consensus_queue.cc:260] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.691893 8404 ts_tablet_manager.cc:1428] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.691854 8426 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:43.691937 8426 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:43.691939 8411 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:43.692001 8411 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:43.692443 8403 tablet_bootstrap.cc:492] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: No bootstrap required, opened a new log
I20260927 21:12:43.692495 8403 ts_tablet_manager.cc:1397] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:43.692777 8403 raft_consensus.cc:348] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.692835 8403 raft_consensus.cc:374] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:43.692847 8403 raft_consensus.cc:729] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcb09f80a5e4253adee88457fe7ce88, State: Initialized, Role: FOLLOWER
I20260927 21:12:43.692880 8403 consensus_queue.cc:260] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.692960 8403 ts_tablet_manager.cc:1428] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:43.693018 8409 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:43.693150 8409 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:43.987874 8409 raft_consensus.cc:482] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:43.987989 8409 raft_consensus.cc:504] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.988117 8409 leader_election.cc:288] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375), 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149)
I20260927 21:12:43.988366 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b7a40aa5fb4fc8b1a05cca713e9870" candidate_uuid: "dfcb09f80a5e4253adee88457fe7ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" is_pre_election: true
I20260927 21:12:43.988456 8049 raft_consensus.cc:2454] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfcb09f80a5e4253adee88457fe7ce88 in term 0.
I20260927 21:12:43.988395 8188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b7a40aa5fb4fc8b1a05cca713e9870" candidate_uuid: "dfcb09f80a5e4253adee88457fe7ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d59d5b52ef4fd1940c789748b61757" is_pre_election: true
I20260927 21:12:43.988520 8188 raft_consensus.cc:2454] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfcb09f80a5e4253adee88457fe7ce88 in term 0.
I20260927 21:12:43.988647 8260 leader_election.cc:302] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfcb09f80a5e4253adee88457fe7ce88, eb0c5c7e4f2d4c149844a6b4957cf6ad; no voters:
I20260927 21:12:43.988749 8409 raft_consensus.cc:2781] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:43.988783 8409 raft_consensus.cc:482] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:43.988802 8409 raft_consensus.cc:3037] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:43.989500 8409 raft_consensus.cc:504] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.989621 8409 leader_election.cc:288] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [CANDIDATE]: Term 1 election: Requested vote from peers eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375), 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149)
I20260927 21:12:43.989818 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b7a40aa5fb4fc8b1a05cca713e9870" candidate_uuid: "dfcb09f80a5e4253adee88457fe7ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad"
I20260927 21:12:43.989835 8188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b7a40aa5fb4fc8b1a05cca713e9870" candidate_uuid: "dfcb09f80a5e4253adee88457fe7ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d59d5b52ef4fd1940c789748b61757"
I20260927 21:12:43.989903 8188 raft_consensus.cc:3037] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:43.989905 8049 raft_consensus.cc:3037] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:43.990473 8188 raft_consensus.cc:2454] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfcb09f80a5e4253adee88457fe7ce88 in term 1.
I20260927 21:12:43.990636 8049 raft_consensus.cc:2454] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfcb09f80a5e4253adee88457fe7ce88 in term 1.
I20260927 21:12:43.990674 8260 leader_election.cc:302] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d59d5b52ef4fd1940c789748b61757, dfcb09f80a5e4253adee88457fe7ce88; no voters:
I20260927 21:12:43.990772 8409 raft_consensus.cc:2781] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:43.990896 8409 raft_consensus.cc:686] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 LEADER]: Becoming Leader. State: Replica: dfcb09f80a5e4253adee88457fe7ce88, State: Running, Role: LEADER
I20260927 21:12:43.990964 8409 consensus_queue.cc:237] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:43.991300 8448 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: TxnStatusTablet state changed. Reason: New leader dfcb09f80a5e4253adee88457fe7ce88. Latest consensus state: current_term: 1 leader_uuid: "dfcb09f80a5e4253adee88457fe7ce88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:43.991392 8448 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:12:43.991593 8450 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:12:43.991699 8450 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:12:43.991659 7891 catalog_manager.cc:5495] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dfcb09f80a5e4253adee88457fe7ce88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } health_report { overall_health: HEALTHY } } }
I20260927 21:12:44.050657 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.047594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46711 (local address 127.0.0.1:51358)
0927 21:12:44.047696 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:44.047711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:44.047765 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:44.047938 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:44.047938 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:44.047940 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:44.047998 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:44.048001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.048674 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.048675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:44.049585 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:44.049588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.049956 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.049957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:44.050030 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.050054 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:44.050430 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:44.050465 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:12:44.050530 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 21:12:44.050584 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:12:44.050679 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.047677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.0.98.3:46711)
0927 21:12:44.047735 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:44.047737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:44.047746 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:44.047847 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:44.047847 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:44.047864 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:44.047920 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:44.048094 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.048582 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.049678 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.049870 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.049933 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.050218 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:44.050341 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:44.050421 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:44.050556 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 21:12:44.050583 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:12:44.051924 8188 raft_consensus.cc:1261] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Refusing update from remote peer dfcb09f80a5e4253adee88457fe7ce88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:44.052062 8049 raft_consensus.cc:1261] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Refusing update from remote peer dfcb09f80a5e4253adee88457fe7ce88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:44.052258 8409 consensus_queue.cc:1035] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:44.052436 8448 consensus_queue.cc:1035] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:44.052919 8411 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: TxnStatusTablet state changed. Reason: New leader dfcb09f80a5e4253adee88457fe7ce88. Latest consensus state: current_term: 1 leader_uuid: "dfcb09f80a5e4253adee88457fe7ce88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:44.053022 8411 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:44.053154 8426 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: TxnStatusTablet state changed. Reason: New leader dfcb09f80a5e4253adee88457fe7ce88. Latest consensus state: current_term: 1 leader_uuid: "dfcb09f80a5e4253adee88457fe7ce88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:44.053244 8426 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:44.054090 8409 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfcb09f80a5e4253adee88457fe7ce88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:44.054181 8409 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:12:44.054343 8426 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfcb09f80a5e4253adee88457fe7ce88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:44.054394 8409 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfcb09f80a5e4253adee88457fe7ce88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:44.054427 8426 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P eb0c5c7e4f2d4c149844a6b4957cf6ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:44.054448 8409 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P dfcb09f80a5e4253adee88457fe7ce88: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:12:44.054792 8411 tablet_replica.cc:439] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfcb09f80a5e4253adee88457fe7ce88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } } }
I20260927 21:12:44.054880 8411 tablet_replica.cc:442] T b2b7a40aa5fb4fc8b1a05cca713e9870 P 74d59d5b52ef4fd1940c789748b61757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:12:44.055783 8456 mvcc.cc:205] Tried to move back new op lower bound from 7334066438354587648 to 7334066438107463680. Current Snapshot: MvccSnapshot[applied={T|T < 7334066438354587648 or (T in {7334066438354587648})}]
I20260927 21:12:44.059160 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.056724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.0.98.2:38149)
0927 21:12:44.056842 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:44.056844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:44.056851 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:12:44.056895 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:44.056896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:44.056913 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:44.056978 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:44.057119 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.057670 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.058408 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.058635 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.058690 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.058842 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:44.058947 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:44.059011 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:44.059052 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 21:12:44.059074 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260927 21:12:44.066258 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.061541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46711 (local address 127.0.98.2:36651)
0927 21:12:44.061690 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:44.061701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:44.061747 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:44.064091 (+ 2344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:44.064093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:44.064095 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:44.064155 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:44.064157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.064856 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.064857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:44.065508 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:44.065511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.065850 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.065850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:44.065908 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.065924 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:44.066086 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:44.066088 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:12:44.066135 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:12:44.066185 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260927 21:12:44.066243 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.063810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36651 (local address 127.0.98.3:46711)
0927 21:12:44.063921 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:44.063923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:44.063939 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:12:44.063964 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:44.063965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:44.063974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:44.064075 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:44.064221 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.064763 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.065589 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.065780 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.065836 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.065989 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:44.066023 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:44.066070 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:44.066140 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 21:12:44.066160 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:12:44.068725 8329 raft_consensus.cc:1261] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Refusing update from remote peer 74d59d5b52ef4fd1940c789748b61757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:44.068756 8049 raft_consensus.cc:1261] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Refusing update from remote peer 74d59d5b52ef4fd1940c789748b61757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:44.068961 8411 consensus_queue.cc:1035] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:44.069132 8437 consensus_queue.cc:1035] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:44.070750 8454 mvcc.cc:205] Tried to move back new op lower bound from 7334066438423531520 to 7334066436631629824. Current Snapshot: MvccSnapshot[applied={T|T < 7334066438423531520}]
I20260927 21:12:44.080564 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.075427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59969 (local address 127.0.98.2:38149)
0927 21:12:44.075469 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:44.075471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:44.077517 (+ 2046us) server_negotiation.cc:407] Connection header received
0927 21:12:44.077577 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:44.077579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:44.077597 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:44.077687 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:44.077968 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.078514 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.079750 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.079979 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.080042 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.080288 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:44.080329 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:44.080380 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:44.080462 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 21:12:44.080485 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:12:44.080940 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.075315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38149 (local address 127.0.98.3:59969)
0927 21:12:44.077415 (+ 2100us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:44.077431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:44.077507 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:44.077717 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:44.077718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:44.077721 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:12:44.077801 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:44.077803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.078619 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.078621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:44.079619 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:44.079622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.080054 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.080055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:44.080136 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.080158 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:12:44.080401 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:12:44.080403 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:12:44.080459 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:12:44.080514 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2051,"thread_start_us":78,"threads_started":1}
I20260927 21:12:44.091804 7891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:44.091922 7891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:44.092573 7891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326: Sending DeleteTablet for 3 replicas of tablet bc1e06d199e84275adf63e8bddc56a52
I20260927 21:12:44.092902 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1e06d199e84275adf63e8bddc56a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45464
I20260927 21:12:44.093117 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1e06d199e84275adf63e8bddc56a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48434
I20260927 21:12:44.093523 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1e06d199e84275adf63e8bddc56a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51354
I20260927 21:12:44.093765 8477 tablet_replica.cc:330] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: stopping tablet replica
I20260927 21:12:44.093859 8477 raft_consensus.cc:2229] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:44.093942 8477 raft_consensus.cc:2258] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:44.094264 8477 ts_tablet_manager.cc:1905] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:44.096436 8477 ts_tablet_manager.cc:1918] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:44.096596 8477 log.cc:1196] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/bc1e06d199e84275adf63e8bddc56a52
I20260927 21:12:44.096920 8477 ts_tablet_manager.cc:1939] T bc1e06d199e84275adf63e8bddc56a52 P dfcb09f80a5e4253adee88457fe7ce88: Deleting consensus metadata
I20260927 21:12:44.097349 7879 catalog_manager.cc:4841] TS dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711): tablet bc1e06d199e84275adf63e8bddc56a52 (table test-table [id=0dde72959e6248e8a0669af22f1669c3]) successfully deleted
I20260927 21:12:44.099028 8475 tablet_replica.cc:330] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: stopping tablet replica
I20260927 21:12:44.099100 8475 raft_consensus.cc:2229] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:44.099227 8476 tablet_replica.cc:330] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: stopping tablet replica
I20260927 21:12:44.099289 8476 raft_consensus.cc:2229] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:44.099339 8476 raft_consensus.cc:2258] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:44.099503 8475 raft_consensus.cc:2258] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:44.099731 8476 ts_tablet_manager.cc:1905] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:44.101457 8475 ts_tablet_manager.cc:1905] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:44.102833 8475 ts_tablet_manager.cc:1918] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:44.102959 8475 log.cc:1196] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/bc1e06d199e84275adf63e8bddc56a52
I20260927 21:12:44.103251 8475 ts_tablet_manager.cc:1939] T bc1e06d199e84275adf63e8bddc56a52 P 74d59d5b52ef4fd1940c789748b61757: Deleting consensus metadata
I20260927 21:12:44.103650 8476 ts_tablet_manager.cc:1918] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:44.103689 7876 catalog_manager.cc:4841] TS 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149): tablet bc1e06d199e84275adf63e8bddc56a52 (table test-table [id=0dde72959e6248e8a0669af22f1669c3]) successfully deleted
I20260927 21:12:44.103935 8476 log.cc:1196] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/bc1e06d199e84275adf63e8bddc56a52
I20260927 21:12:44.104208 8476 ts_tablet_manager.cc:1939] T bc1e06d199e84275adf63e8bddc56a52 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Deleting consensus metadata
I20260927 21:12:44.104574 7876 catalog_manager.cc:4841] TS eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375): tablet bc1e06d199e84275adf63e8bddc56a52 (table test-table [id=0dde72959e6248e8a0669af22f1669c3]) successfully deleted
Sep 27 21:12:44 dist-test-slave-r9x1 krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:12:44 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:44 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:12:44.122351 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:44.117108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.0.98.62:46835)
0927 21:12:44.117158 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:44.117161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:44.117314 (+ 153us) server_negotiation.cc:407] Connection header received
0927 21:12:44.117398 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:44.117399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:44.117410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:44.117456 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:44.118023 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.118526 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.119262 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.119451 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.119505 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.122280 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50888: BlockingRecv error: recv got EOF from 127.0.0.1:50888 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 21:12:44 dist-test-slave-r9x1 krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:12:44.130321 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.123199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.0.98.62:46835)
0927 21:12:44.123240 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:44.123242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:44.123439 (+ 197us) server_negotiation.cc:407] Connection header received
0927 21:12:44.123520 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:44.123521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:44.123531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:44.123579 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:44.124035 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.124585 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.125251 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.125483 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.125539 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.127874 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:44.127875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:44.127875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:44.127883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:44.129259 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:44.129597 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:44.129598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:44.129598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:44.129651 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:44.129823 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:44.129824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:44.129824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:44.129936 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:44.130018 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:44.130139 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:12:44.130226 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:12:44.132109 7891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:12:44.132272 7891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:44.133527 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 4038f9155ca84f2a938c871d689e2799 (DEFAULT_TABLE table=test-table [id=e576c64656ec4539b6b7d2533bc4b33c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:44.133664 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4038f9155ca84f2a938c871d689e2799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:44.134811 8405 tablet_bootstrap.cc:492] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Bootstrap starting.
I20260927 21:12:44.134822 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 4038f9155ca84f2a938c871d689e2799 (DEFAULT_TABLE table=test-table [id=e576c64656ec4539b6b7d2533bc4b33c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:44.134925 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4038f9155ca84f2a938c871d689e2799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:44.135161 8405 tablet_bootstrap.cc:654] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:44.135403 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 4038f9155ca84f2a938c871d689e2799 (DEFAULT_TABLE table=test-table [id=e576c64656ec4539b6b7d2533bc4b33c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:44.135507 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4038f9155ca84f2a938c871d689e2799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:44.136219 8403 tablet_bootstrap.cc:492] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: Bootstrap starting.
I20260927 21:12:44.136468 8403 tablet_bootstrap.cc:654] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:44.136775 8404 tablet_bootstrap.cc:492] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: Bootstrap starting.
I20260927 21:12:44.137188 8404 tablet_bootstrap.cc:654] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:44.137849 8405 tablet_bootstrap.cc:492] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: No bootstrap required, opened a new log
I20260927 21:12:44.137856 8404 tablet_bootstrap.cc:492] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: No bootstrap required, opened a new log
I20260927 21:12:44.137899 8404 ts_tablet_manager.cc:1397] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:44.137899 8405 ts_tablet_manager.cc:1397] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:12:44.138020 8405 raft_consensus.cc:348] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.138051 8404 raft_consensus.cc:348] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.138091 8405 raft_consensus.cc:374] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:44.138108 8404 raft_consensus.cc:374] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:44.138127 8404 raft_consensus.cc:729] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d59d5b52ef4fd1940c789748b61757, State: Initialized, Role: FOLLOWER
I20260927 21:12:44.138110 8405 raft_consensus.cc:729] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0c5c7e4f2d4c149844a6b4957cf6ad, State: Initialized, Role: FOLLOWER
I20260927 21:12:44.138166 8405 consensus_queue.cc:260] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.138166 8404 consensus_queue.cc:260] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.138252 8405 ts_tablet_manager.cc:1428] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:44.138262 8404 ts_tablet_manager.cc:1428] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:44.138640 8403 tablet_bootstrap.cc:492] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: No bootstrap required, opened a new log
I20260927 21:12:44.138689 8403 ts_tablet_manager.cc:1397] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 21:12:44.138789 8403 raft_consensus.cc:348] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.138850 8403 raft_consensus.cc:374] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:44.138867 8403 raft_consensus.cc:729] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcb09f80a5e4253adee88457fe7ce88, State: Initialized, Role: FOLLOWER
I20260927 21:12:44.138904 8403 consensus_queue.cc:260] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.138989 8403 ts_tablet_manager.cc:1428] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:44.331188 8469 raft_consensus.cc:482] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:44.331298 8469 raft_consensus.cc:504] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.331452 8469 leader_election.cc:288] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375), dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711)
I20260927 21:12:44.331727 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4038f9155ca84f2a938c871d689e2799" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb09f80a5e4253adee88457fe7ce88" is_pre_election: true
I20260927 21:12:44.331744 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4038f9155ca84f2a938c871d689e2799" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" is_pre_election: true
I20260927 21:12:44.331828 8329 raft_consensus.cc:2454] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 0.
I20260927 21:12:44.331830 8049 raft_consensus.cc:2454] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 0.
I20260927 21:12:44.332000 8120 leader_election.cc:302] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d59d5b52ef4fd1940c789748b61757, eb0c5c7e4f2d4c149844a6b4957cf6ad; no voters:
I20260927 21:12:44.332103 8469 raft_consensus.cc:2781] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:44.332144 8469 raft_consensus.cc:482] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:44.332165 8469 raft_consensus.cc:3037] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:44.332880 8469 raft_consensus.cc:504] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.333052 8469 leader_election.cc:288] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 election: Requested vote from peers eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375), dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711)
I20260927 21:12:44.333184 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4038f9155ca84f2a938c871d689e2799" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad"
I20260927 21:12:44.333258 8049 raft_consensus.cc:3037] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:44.333254 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4038f9155ca84f2a938c871d689e2799" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb09f80a5e4253adee88457fe7ce88"
I20260927 21:12:44.333371 8329 raft_consensus.cc:3037] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:44.333885 8049 raft_consensus.cc:2454] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 1.
I20260927 21:12:44.334043 8120 leader_election.cc:302] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d59d5b52ef4fd1940c789748b61757, eb0c5c7e4f2d4c149844a6b4957cf6ad; no voters:
I20260927 21:12:44.334064 8329 raft_consensus.cc:2454] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 1.
I20260927 21:12:44.334137 8469 raft_consensus.cc:2781] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:44.334187 8469 raft_consensus.cc:686] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 1 LEADER]: Becoming Leader. State: Replica: 74d59d5b52ef4fd1940c789748b61757, State: Running, Role: LEADER
I20260927 21:12:44.334237 8469 consensus_queue.cc:237] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.334817 7892 catalog_manager.cc:5495] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74d59d5b52ef4fd1940c789748b61757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } health_report { overall_health: UNKNOWN } } }
I20260927 21:12:44.366190 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.363945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.0.98.2:38149)
0927 21:12:44.363991 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:44.363993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:44.364001 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:44.364025 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:44.364026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:44.364057 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:44.364121 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:12:44.364240 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.364725 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.365593 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:44.365792 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:44.365855 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:44.366001 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:12:44.366028 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:12:44.366069 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:44.366104 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 21:12:44.366128 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:12:44.366971 8329 raft_consensus.cc:1261] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Refusing update from remote peer 74d59d5b52ef4fd1940c789748b61757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:44.367020 8049 raft_consensus.cc:1261] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Refusing update from remote peer 74d59d5b52ef4fd1940c789748b61757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:44.367218 8469 consensus_queue.cc:1035] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:44.367327 8411 consensus_queue.cc:1035] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:44.369838 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:44.369930 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:44.370545 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326: Sending DeleteTablet for 3 replicas of tablet 4038f9155ca84f2a938c871d689e2799
I20260927 21:12:44.370795 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 4038f9155ca84f2a938c871d689e2799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51354
I20260927 21:12:44.370810 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 4038f9155ca84f2a938c871d689e2799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45464
I20260927 21:12:44.370890 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4038f9155ca84f2a938c871d689e2799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48434
I20260927 21:12:44.370977 8477 tablet_replica.cc:330] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: stopping tablet replica
I20260927 21:12:44.370983 8476 tablet_replica.cc:330] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: stopping tablet replica
I20260927 21:12:44.371028 8477 raft_consensus.cc:2229] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:44.371037 8476 raft_consensus.cc:2229] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:44.371069 8477 raft_consensus.cc:2258] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:44.371074 8476 raft_consensus.cc:2258] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:44.371038 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:12:44.371145 8475 tablet_replica.cc:330] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: stopping tablet replica
W20260927 21:12:44.371182 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:12:44.371194 8475 raft_consensus.cc:2229] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:44.371250 8475 raft_consensus.cc:2258] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:44.371417 8476 ts_tablet_manager.cc:1905] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:44.371690 8477 ts_tablet_manager.cc:1905] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:44.371733 8475 ts_tablet_manager.cc:1905] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:44.372361 8029 tablet_service.cc:1467] Processing CreateTablet for tablet b31ff9e700734f13a86e61582507aec7 (DEFAULT_TABLE table=test-table [id=a546b6d896174ffe8860b352fecef586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:44.372464 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31ff9e700734f13a86e61582507aec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:44.373116 8169 tablet_service.cc:1467] Processing CreateTablet for tablet b31ff9e700734f13a86e61582507aec7 (DEFAULT_TABLE table=test-table [id=a546b6d896174ffe8860b352fecef586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:44.373226 8475 ts_tablet_manager.cc:1918] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:44.373227 8476 ts_tablet_manager.cc:1918] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:44.373227 8477 ts_tablet_manager.cc:1918] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:12:44.373296 8475 log.cc:1196] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/4038f9155ca84f2a938c871d689e2799
I20260927 21:12:44.373301 8477 log.cc:1196] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/4038f9155ca84f2a938c871d689e2799
I20260927 21:12:44.373301 8476 log.cc:1196] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/4038f9155ca84f2a938c871d689e2799
I20260927 21:12:44.373507 8476 ts_tablet_manager.cc:1939] T 4038f9155ca84f2a938c871d689e2799 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Deleting consensus metadata
I20260927 21:12:44.373507 8477 ts_tablet_manager.cc:1939] T 4038f9155ca84f2a938c871d689e2799 P dfcb09f80a5e4253adee88457fe7ce88: Deleting consensus metadata
I20260927 21:12:44.373505 8475 ts_tablet_manager.cc:1939] T 4038f9155ca84f2a938c871d689e2799 P 74d59d5b52ef4fd1940c789748b61757: Deleting consensus metadata
I20260927 21:12:44.373771 8309 tablet_service.cc:1467] Processing CreateTablet for tablet b31ff9e700734f13a86e61582507aec7 (DEFAULT_TABLE table=test-table [id=a546b6d896174ffe8860b352fecef586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:12:44.373230 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31ff9e700734f13a86e61582507aec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:44.373816 7876 catalog_manager.cc:4841] TS 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149): tablet 4038f9155ca84f2a938c871d689e2799 (table test-table [id=e576c64656ec4539b6b7d2533bc4b33c]) successfully deleted
I20260927 21:12:44.373860 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31ff9e700734f13a86e61582507aec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:44.374131 8405 tablet_bootstrap.cc:492] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Bootstrap starting.
I20260927 21:12:44.374197 7876 catalog_manager.cc:4841] TS eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375): tablet 4038f9155ca84f2a938c871d689e2799 (table test-table [id=e576c64656ec4539b6b7d2533bc4b33c]) successfully deleted
I20260927 21:12:44.374416 8405 tablet_bootstrap.cc:654] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:44.374918 8403 tablet_bootstrap.cc:492] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: Bootstrap starting.
I20260927 21:12:44.374943 8405 tablet_bootstrap.cc:492] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad: No bootstrap required, opened a new log
I20260927 21:12:44.374976 8405 ts_tablet_manager.cc:1397] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:44.375082 8405 raft_consensus.cc:348] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.375159 8405 raft_consensus.cc:374] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:44.375180 8405 raft_consensus.cc:729] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0c5c7e4f2d4c149844a6b4957cf6ad, State: Initialized, Role: FOLLOWER
I20260927 21:12:44.375186 8403 tablet_bootstrap.cc:654] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:44.375229 8405 consensus_queue.cc:260] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.375325 8405 ts_tablet_manager.cc:1428] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:44.375402 8404 tablet_bootstrap.cc:492] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: Bootstrap starting.
I20260927 21:12:44.375682 8404 tablet_bootstrap.cc:654] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:44.375921 7879 catalog_manager.cc:4841] TS dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711): tablet 4038f9155ca84f2a938c871d689e2799 (table test-table [id=e576c64656ec4539b6b7d2533bc4b33c]) successfully deleted
I20260927 21:12:44.376156 8404 tablet_bootstrap.cc:492] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: No bootstrap required, opened a new log
I20260927 21:12:44.376196 8404 ts_tablet_manager.cc:1397] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 21:12:44.376281 8404 raft_consensus.cc:348] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.376339 8404 raft_consensus.cc:374] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:44.376351 8404 raft_consensus.cc:729] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d59d5b52ef4fd1940c789748b61757, State: Initialized, Role: FOLLOWER
I20260927 21:12:44.376381 8404 consensus_queue.cc:260] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.376457 8404 ts_tablet_manager.cc:1428] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:44.376683 8403 tablet_bootstrap.cc:492] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: No bootstrap required, opened a new log
I20260927 21:12:44.376730 8403 ts_tablet_manager.cc:1397] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:12:44.376830 8403 raft_consensus.cc:348] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.376886 8403 raft_consensus.cc:374] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:44.376899 8403 raft_consensus.cc:729] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcb09f80a5e4253adee88457fe7ce88, State: Initialized, Role: FOLLOWER
I20260927 21:12:44.376930 8403 consensus_queue.cc:260] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.377007 8403 ts_tablet_manager.cc:1428] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:44.478927 8469 raft_consensus.cc:482] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:12:44.479036 8469 raft_consensus.cc:504] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.479180 8469 leader_election.cc:288] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375), dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711)
I20260927 21:12:44.479425 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31ff9e700734f13a86e61582507aec7" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" is_pre_election: true
I20260927 21:12:44.479450 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31ff9e700734f13a86e61582507aec7" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb09f80a5e4253adee88457fe7ce88" is_pre_election: true
I20260927 21:12:44.479512 8049 raft_consensus.cc:2454] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 0.
I20260927 21:12:44.479550 8329 raft_consensus.cc:2454] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 0.
I20260927 21:12:44.479676 8120 leader_election.cc:302] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d59d5b52ef4fd1940c789748b61757, eb0c5c7e4f2d4c149844a6b4957cf6ad; no voters:
I20260927 21:12:44.479773 8469 raft_consensus.cc:2781] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:12:44.479816 8469 raft_consensus.cc:482] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:12:44.479830 8469 raft_consensus.cc:3037] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:44.480439 8469 raft_consensus.cc:504] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.480551 8469 leader_election.cc:288] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 election: Requested vote from peers eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375), dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711)
I20260927 21:12:44.480710 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31ff9e700734f13a86e61582507aec7" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad"
I20260927 21:12:44.480742 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31ff9e700734f13a86e61582507aec7" candidate_uuid: "74d59d5b52ef4fd1940c789748b61757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcb09f80a5e4253adee88457fe7ce88"
I20260927 21:12:44.480791 8049 raft_consensus.cc:3037] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:44.480809 8329 raft_consensus.cc:3037] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:44.481570 8329 raft_consensus.cc:2454] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 1.
I20260927 21:12:44.481570 8049 raft_consensus.cc:2454] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74d59d5b52ef4fd1940c789748b61757 in term 1.
I20260927 21:12:44.481743 8123 leader_election.cc:302] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74d59d5b52ef4fd1940c789748b61757, dfcb09f80a5e4253adee88457fe7ce88; no voters:
I20260927 21:12:44.481858 8469 raft_consensus.cc:2781] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:44.481921 8469 raft_consensus.cc:686] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 1 LEADER]: Becoming Leader. State: Replica: 74d59d5b52ef4fd1940c789748b61757, State: Running, Role: LEADER
I20260927 21:12:44.481971 8469 consensus_queue.cc:237] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } }
I20260927 21:12:44.482547 7891 catalog_manager.cc:5495] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74d59d5b52ef4fd1940c789748b61757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74d59d5b52ef4fd1940c789748b61757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 } health_report { overall_health: UNKNOWN } } }
I20260927 21:12:44.497704 8049 raft_consensus.cc:1261] T b31ff9e700734f13a86e61582507aec7 P eb0c5c7e4f2d4c149844a6b4957cf6ad [term 1 FOLLOWER]: Refusing update from remote peer 74d59d5b52ef4fd1940c789748b61757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:44.497751 8329 raft_consensus.cc:1261] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Refusing update from remote peer 74d59d5b52ef4fd1940c789748b61757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:12:44.497960 8469 consensus_queue.cc:1035] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0c5c7e4f2d4c149844a6b4957cf6ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:44.498072 8437 consensus_queue.cc:1035] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcb09f80a5e4253adee88457fe7ce88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:12:44.514216 7890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:44.514341 7890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:12:44.514945 7890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 687d2e97a8c64094bd6e87c3e299e326: Sending DeleteTablet for 3 replicas of tablet b31ff9e700734f13a86e61582507aec7
I20260927 21:12:44.515218 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet b31ff9e700734f13a86e61582507aec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48434
I20260927 21:12:44.515219 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet b31ff9e700734f13a86e61582507aec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45464
I20260927 21:12:44.515331 8475 tablet_replica.cc:330] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: stopping tablet replica
I20260927 21:12:44.515386 8475 raft_consensus.cc:2229] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:12:44.515446 8475 raft_consensus.cc:2258] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:44.515488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 7964
I20260927 21:12:44.515966 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet b31ff9e700734f13a86e61582507aec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:12:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51354
I20260927 21:12:44.516099 8477 tablet_replica.cc:330] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: stopping tablet replica
I20260927 21:12:44.516151 8477 raft_consensus.cc:2229] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:12:44.516188 8477 raft_consensus.cc:2258] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:12:44.516500 8475 ts_tablet_manager.cc:1905] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:44.516908 8477 ts_tablet_manager.cc:1905] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:12:44.518217 8475 ts_tablet_manager.cc:1918] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:44.518293 8475 log.cc:1196] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/b31ff9e700734f13a86e61582507aec7
I20260927 21:12:44.518500 8475 ts_tablet_manager.cc:1939] T b31ff9e700734f13a86e61582507aec7 P 74d59d5b52ef4fd1940c789748b61757: Deleting consensus metadata
I20260927 21:12:44.518646 8477 ts_tablet_manager.cc:1918] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:12:44.518712 8477 log.cc:1196] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/b31ff9e700734f13a86e61582507aec7
I20260927 21:12:44.518905 8477 ts_tablet_manager.cc:1939] T b31ff9e700734f13a86e61582507aec7 P dfcb09f80a5e4253adee88457fe7ce88: Deleting consensus metadata
I20260927 21:12:44.518981 7876 catalog_manager.cc:4841] TS 74d59d5b52ef4fd1940c789748b61757 (127.0.98.2:38149): tablet b31ff9e700734f13a86e61582507aec7 (table test-table [id=a546b6d896174ffe8860b352fecef586]) successfully deleted
I20260927 21:12:44.522446 7879 catalog_manager.cc:4841] TS dfcb09f80a5e4253adee88457fe7ce88 (127.0.98.3:46711): tablet b31ff9e700734f13a86e61582507aec7 (table test-table [id=a546b6d896174ffe8860b352fecef586]) successfully deleted
W20260927 21:12:44.524559 7876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34375 (error 108)
I20260927 21:12:44.524852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 8104
I20260927 21:12:44.525826 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:44.524917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34375 (local address 127.0.0.1:48452)
0927 21:12:44.525717 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:44.525776 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":39,"threads_started":1}
W20260927 21:12:44.526335 7876 catalog_manager.cc:4584] TS eb0c5c7e4f2d4c149844a6b4957cf6ad (127.0.98.1:34375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b31ff9e700734f13a86e61582507aec7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34375: connect: Connection refused (error 111)
I20260927 21:12:44.532404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 8244
I20260927 21:12:44.538137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 7860
2026-09-27T21:12:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3023 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:44 dist-test-slave-r9x1 krb5kdc[8507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:44 dist-test-slave-r9x1 krb5kdc[8507](info): set up 2 sockets
Sep 27 21:12:44 dist-test-slave-r9x1 krb5kdc[8507](info): commencing operation
krb5kdc: starting...
W20260927 21:12:45.501603 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:45 dist-test-slave-r9x1 krb5kdc[8507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:45.620330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43351
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:45.702905 8520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:45.703140 8520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:45.703164 8520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:45.704672 8520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:45.704794 8520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:45.704810 8520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:45.704823 8520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:45.704836 8520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:45.707472 8520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:45.707767 8520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:45.708052 8520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:45.713048 8528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:45.713080 8520 server_base.cc:1034] running on GCE node
W20260927 21:12:45.713048 8525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:45.713047 8526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:45.713662 8520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:45.713979 8520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:45.715153 8520 hybrid_clock.cc:648] HybridClock initialized: now 1790543565715134 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T21:12:45Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1180 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:12:45 dist-test-slave-r9x1 krb5kdc[8533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:12:45 dist-test-slave-r9x1 krb5kdc[8533](info): set up 2 sockets
Sep 27 21:12:45 dist-test-slave-r9x1 krb5kdc[8533](info): commencing operation
krb5kdc: starting...
W20260927 21:12:46.681631 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:12:46 dist-test-slave-r9x1 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:12:46.804641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40661
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:46.885422 8546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:46.885644 8546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:46.885676 8546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:46.886920 8546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:46.886974 8546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:46.886992 8546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:46.887008 8546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:46.887022 8546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:46.888623 8546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:46.888893 8546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:46.889109 8546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:46.893941 8552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:46.893935 8551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:46.893941 8554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:46.894089 8546 server_base.cc:1034] running on GCE node
I20260927 21:12:46.894512 8546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:46.894824 8546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:46.895987 8546 hybrid_clock.cc:648] HybridClock initialized: now 1790543566895967 us; error 37 us; skew 500 ppm
Sep 27 21:12:46 dist-test-slave-r9x1 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:46.898502 8546 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:46.898983 8546 webserver.cc:460] Webserver started at http://127.0.98.62:35689/ using document root <none> and password file <none>
I20260927 21:12:46.899189 8546 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:46.899240 8546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:46.899334 8546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:46.900388 8546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb"
format_stamp: "Formatted at 2026-09-27 21:12:46 on dist-test-slave-r9x1"
server_key: "81311e71b21482e9e7b81fe7764db821"
server_key_iv: "d4927be1eed1ef0d0351c00510489a45"
server_key_version: "encryptionkey@0"
I20260927 21:12:46.900683 8546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb"
format_stamp: "Formatted at 2026-09-27 21:12:46 on dist-test-slave-r9x1"
server_key: "81311e71b21482e9e7b81fe7764db821"
server_key_iv: "d4927be1eed1ef0d0351c00510489a45"
server_key_version: "encryptionkey@0"
I20260927 21:12:46.902032 8546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:12:46.902740 8561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:46.902958 8546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:46.903052 8546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb"
format_stamp: "Formatted at 2026-09-27 21:12:46 on dist-test-slave-r9x1"
server_key: "81311e71b21482e9e7b81fe7764db821"
server_key_iv: "d4927be1eed1ef0d0351c00510489a45"
server_key_version: "encryptionkey@0"
I20260927 21:12:46.903123 8546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:46.916172 8546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:46.918411 8546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:46.918532 8546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:46.922714 8546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40661
I20260927 21:12:46.922756 8613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40661 every 8 connection(s)
I20260927 21:12:46.923280 8546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:46.923970 8614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:12:46.926486 8614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: Bootstrap starting.
I20260927 21:12:46.927124 8614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: Neither blocks nor log segments found. Creating new log.
I20260927 21:12:46.927413 8614 log.cc:824] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:46.928284 8614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: No bootstrap required, opened a new log
I20260927 21:12:46.929665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 8546
I20260927 21:12:46.929764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:12:46.929951 392 external_mini_cluster.cc:1428] Setting key ab1b345b983ea8c3cd9235cd5c67920b
I20260927 21:12:46.930337 8614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } }
I20260927 21:12:46.930557 8614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:12:46.930599 8614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8f62e1f2c64b769d6ee58602eb7bfb, State: Initialized, Role: FOLLOWER
I20260927 21:12:46.930748 8614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } }
I20260927 21:12:46.930821 8614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:46.930853 8614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:46.930905 8614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:12:46.931715 8614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } }
I20260927 21:12:46.931844 8614 leader_election.cc:302] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d8f62e1f2c64b769d6ee58602eb7bfb; no voters:
I20260927 21:12:46.932005 8614 leader_election.cc:288] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:12:46.932073 8619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:12:46.932276 8614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:46.932292 8619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 1 LEADER]: Becoming Leader. State: Replica: 3d8f62e1f2c64b769d6ee58602eb7bfb, State: Running, Role: LEADER
I20260927 21:12:46.932411 8619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } }
I20260927 21:12:46.932776 8620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } } }
I20260927 21:12:46.932861 8620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:46.932986 8620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d8f62e1f2c64b769d6ee58602eb7bfb. Latest consensus state: current_term: 1 leader_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } } }
I20260927 21:12:46.933029 8620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:46.933049 8628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:46.933676 8628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:46.935964 8628 catalog_manager.cc:1347] Generated new cluster ID: 3362c42d445f461eba43b243661c74c5
I20260927 21:12:46.936026 8628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:12:46 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:46.942173 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:46.930545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.0.98.62:40661)
0927 21:12:46.931193 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:46.931204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:46.931244 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:12:46.931380 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:46.931388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:46.931449 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:46.931615 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:46.932484 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:46.933960 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:46.934744 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:46.935360 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:46.935465 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:46.938329 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:46.938334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:46.938335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:46.938349 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:46.940296 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:46.940763 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:46.940766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:46.940767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:46.940889 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:46.941094 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:46.941094 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:46.941095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:46.941424 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:46.941590 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:46.941695 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:12:46.941946 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":536,"threads_started":1}
I20260927 21:12:46.954638 8628 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:12:46.955180 8628 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:46.958773 8628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: Generated new TSK 0
I20260927 21:12:46.958971 8628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:12:46.988761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40661
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:47.072435 8640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:47.072664 8640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:47.072695 8640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:47.073967 8640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:47.074041 8640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:47.074070 8640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:47.075744 8640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:47.076016 8640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:47.076254 8640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:47.081425 8648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:47.081427 8645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:47.081446 8640 server_base.cc:1034] running on GCE node
W20260927 21:12:47.081429 8646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:47.081883 8640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:47.082131 8640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:47.083302 8640 hybrid_clock.cc:648] HybridClock initialized: now 1790543567083277 us; error 33 us; skew 500 ppm
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:47.085901 8640 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:12:47.086356 8640 webserver.cc:460] Webserver started at http://127.0.98.1:44379/ using document root <none> and password file <none>
I20260927 21:12:47.086542 8640 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:47.086583 8640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:47.086683 8640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:47.087730 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "5c72349a9b8b45a2aa789c1a9471c9f0"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "66fda849da4dfafd1d2d6f6e8068ef56"
server_key_iv: "eaa8095931b530b62f4222aca37531ad"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.088042 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5c72349a9b8b45a2aa789c1a9471c9f0"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "66fda849da4dfafd1d2d6f6e8068ef56"
server_key_iv: "eaa8095931b530b62f4222aca37531ad"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.089538 8640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:12:47.090387 8655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.090571 8640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:47.090653 8640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "5c72349a9b8b45a2aa789c1a9471c9f0"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "66fda849da4dfafd1d2d6f6e8068ef56"
server_key_iv: "eaa8095931b530b62f4222aca37531ad"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.090734 8640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:47.103304 8640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:47.106297 8640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:47.106436 8640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:47.106683 8640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:47.107079 8640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:47.107118 8640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.107149 8640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:47.107170 8640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.114082 8640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41013
I20260927 21:12:47.114122 8768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41013 every 8 connection(s)
I20260927 21:12:47.114464 8640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:47.123929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 8640
I20260927 21:12:47.124075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:12:47.124310 392 external_mini_cluster.cc:1428] Setting key 4cd78263f067d0d737074544aa42c57c
I20260927 21:12:47.125495 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.115288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43505 (local address 127.0.98.62:40661)
0927 21:12:47.115346 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:47.115348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:47.115545 (+ 197us) server_negotiation.cc:407] Connection header received
0927 21:12:47.116520 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:47.116521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:47.116539 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:47.116619 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:47.117385 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.117892 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.119409 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.119617 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.119679 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.121996 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:47.121998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:47.121998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:47.122008 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:47.123331 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:47.123901 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:47.123905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:47.123905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:47.124039 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:47.124583 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:47.124585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:47.124586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:47.124776 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:47.124840 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:47.125228 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 21:12:47.125383 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:12:47.125661 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.115210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:43505)
0927 21:12:47.115377 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:47.115424 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:47.116366 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:47.116751 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:47.116755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:47.117041 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:47.117244 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.117249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.118082 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.118084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.119295 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.119301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.119730 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.119732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.119878 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.120427 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:47.120437 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:47.121885 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:47.123459 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:47.123468 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:47.123471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:47.123756 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:47.124206 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:47.124208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:47.124209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:47.124458 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:47.124846 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:47.124866 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:47.125075 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 21:12:47.125193 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 21:12:47.125411 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260927 21:12:47.126209 8769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40661
I20260927 21:12:47.126336 8769 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:47.126564 8769 heartbeater.cc:511] Master 127.0.98.62:40661 requested a full tablet report, sending...
I20260927 21:12:47.127177 8576 ts_manager.cc:194] Registered new tserver with Master: 5c72349a9b8b45a2aa789c1a9471c9f0 (127.0.98.1:41013)
I20260927 21:12:47.127887 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43505
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:12:47.152279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40661
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:47.236020 8774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:47.236264 8774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:47.236303 8774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:47.237708 8774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:47.237757 8774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:47.237797 8774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:47.239526 8774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:47.239809 8774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:47.240038 8774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:47.245110 8780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:47.245097 8779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:47.245091 8782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:47.245404 8774 server_base.cc:1034] running on GCE node
I20260927 21:12:47.245606 8774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:47.245838 8774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:47.247013 8774 hybrid_clock.cc:648] HybridClock initialized: now 1790543567247001 us; error 41 us; skew 500 ppm
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:47.249610 8774 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:12:47.250062 8774 webserver.cc:460] Webserver started at http://127.0.98.2:40651/ using document root <none> and password file <none>
I20260927 21:12:47.250267 8774 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:47.250311 8774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:47.250418 8774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:47.251504 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "5f487c2318f3407d8a519408aebde94e"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "c9f7a4d9794a8e0c6585687d112574e1"
server_key_iv: "95f89b8685d3e68233d7a04343e1992e"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.251818 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5f487c2318f3407d8a519408aebde94e"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "c9f7a4d9794a8e0c6585687d112574e1"
server_key_iv: "95f89b8685d3e68233d7a04343e1992e"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.253125 8774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:12:47.253960 8789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.254161 8774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:47.254279 8774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "5f487c2318f3407d8a519408aebde94e"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "c9f7a4d9794a8e0c6585687d112574e1"
server_key_iv: "95f89b8685d3e68233d7a04343e1992e"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.254354 8774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:47.270071 8774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:47.272217 8774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:47.272353 8774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:47.272595 8774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:47.272894 8774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:47.272933 8774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.272955 8774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:47.272979 8774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.278661 8774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41173
I20260927 21:12:47.278729 8902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41173 every 8 connection(s)
I20260927 21:12:47.279021 8774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:47.287971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 8774
I20260927 21:12:47.288079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:12:47.288194 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.279812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41847 (local address 127.0.98.62:40661)
0927 21:12:47.279873 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:47.279875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:47.280054 (+ 179us) server_negotiation.cc:407] Connection header received
0927 21:12:47.281001 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:47.281002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:47.281015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:47.281063 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:47.281745 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.282250 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.283236 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.283430 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.283485 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.285702 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:47.285704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:47.285704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:47.285711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:47.287002 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:47.287407 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:47.287408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:47.287408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:47.287454 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:47.287638 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:47.287638 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:47.287638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:47.287738 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:47.287785 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:47.288018 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 21:12:47.288107 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:12:47.288304 392 external_mini_cluster.cc:1428] Setting key e3dd8ef35360a4264faf42573b0f5ecb
I20260927 21:12:47.288380 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.279758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:41847)
0927 21:12:47.279939 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:47.279975 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:47.280891 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:47.281133 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:47.281138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:47.281455 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:47.281654 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.281660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.282342 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.282343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.283143 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.283145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.283500 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.283501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.283624 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.284133 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:47.284141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:47.285601 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:47.287089 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:47.287096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:47.287098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:47.287343 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:47.287501 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:47.287501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:47.287502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:47.287596 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:47.287779 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:47.287782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:47.287903 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 21:12:47.287983 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:12:47.288145 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260927 21:12:47.288885 8903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40661
I20260927 21:12:47.288988 8903 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:47.289230 8903 heartbeater.cc:511] Master 127.0.98.62:40661 requested a full tablet report, sending...
I20260927 21:12:47.289651 8576 ts_manager.cc:194] Registered new tserver with Master: 5f487c2318f3407d8a519408aebde94e (127.0.98.2:41173)
I20260927 21:12:47.290010 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41847
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:12:47.314584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40661
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:47.395983 8908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:47.396203 8908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:47.396229 8908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:47.397491 8908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:47.397538 8908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:47.397567 8908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:47.399164 8908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:47.399412 8908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:47.399641 8908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:47.404819 8914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:47.404781 8913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:47.404775 8916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:47.404852 8908 server_base.cc:1034] running on GCE node
I20260927 21:12:47.405231 8908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:47.405524 8908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:47.406704 8908 hybrid_clock.cc:648] HybridClock initialized: now 1790543567406682 us; error 38 us; skew 500 ppm
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:47.409221 8908 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:12:47.409739 8908 webserver.cc:460] Webserver started at http://127.0.98.3:43665/ using document root <none> and password file <none>
I20260927 21:12:47.409958 8908 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:47.410004 8908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:47.410112 8908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:12:47.411175 8908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "e61be972eac341a28117e967f77073d9"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "11b7eaa85b124b6496e95b30d73d1386"
server_key_iv: "479780c85b9dd4b90c8ad1c5f899d747"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.411513 8908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e61be972eac341a28117e967f77073d9"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "11b7eaa85b124b6496e95b30d73d1386"
server_key_iv: "479780c85b9dd4b90c8ad1c5f899d747"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.412791 8908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:12:47.413579 8923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.413733 8908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:47.413806 8908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "e61be972eac341a28117e967f77073d9"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "11b7eaa85b124b6496e95b30d73d1386"
server_key_iv: "479780c85b9dd4b90c8ad1c5f899d747"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.413878 8908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:47.438572 8908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:47.440534 8908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:47.440658 8908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:47.440905 8908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:47.441220 8908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:47.441253 8908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.441318 8908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:47.441345 8908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.447224 8908 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37235
I20260927 21:12:47.447274 9036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37235 every 8 connection(s)
I20260927 21:12:47.447595 8908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:12:47.450440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 8908
I20260927 21:12:47.450534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:12:47.450677 392 external_mini_cluster.cc:1428] Setting key 3b9dc0827138614ebcc3711afd1739ac
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:12:47.457254 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.448293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53575 (local address 127.0.98.62:40661)
0927 21:12:47.448358 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:47.448360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:47.448545 (+ 185us) server_negotiation.cc:407] Connection header received
0927 21:12:47.449614 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:47.449616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:47.449628 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:47.449679 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:47.450351 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.450991 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.452148 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.452362 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.452418 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.454567 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:47.454568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:47.454568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:47.454576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:47.455845 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:47.456295 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:47.456297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:47.456297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:47.456358 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:47.456642 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:47.456643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:47.456643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:47.456766 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:47.456818 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:47.457112 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 21:12:47.457184 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:12:47.457566 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.448253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:53575)
0927 21:12:47.448425 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:47.448468 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:47.449502 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:47.449751 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:47.449756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:47.450044 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:47.450236 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.450240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.451083 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.451085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.452071 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.452073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.452433 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.452434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.452640 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.453111 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:47.453119 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:47.454476 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:12:47.455943 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:47.455950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:47.455952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:47.456195 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:47.456450 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:12:47.456451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:12:47.456452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:12:47.456586 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:12:47.456842 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:12:47.456848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:12:47.456991 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 21:12:47.457088 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 21:12:47.457225 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260927 21:12:47.458051 9037 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40661
I20260927 21:12:47.458149 9037 heartbeater.cc:464] Registering TS with master...
I20260927 21:12:47.458338 9037 heartbeater.cc:511] Master 127.0.98.62:40661 requested a full tablet report, sending...
I20260927 21:12:47.458698 8576 ts_manager.cc:194] Registered new tserver with Master: e61be972eac341a28117e967f77073d9 (127.0.98.3:37235)
I20260927 21:12:47.459118 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53575
I20260927 21:12:47.462512 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 21:12:47.535820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 8640
I20260927 21:12:47.539853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 8774
I20260927 21:12:47.543880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 8908
I20260927 21:12:47.547834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 8546
I20260927 21:12:47.552248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40661
--webserver_interface=127.0.98.62
--webserver_port=35689
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:12:47.634467 9046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:47.634687 9046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:47.634709 9046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:47.635874 9046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:12:47.635915 9046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:47.635928 9046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:47.635938 9046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:12:47.635950 9046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:12:47.637472 9046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:47.637676 9046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:47.637862 9046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:47.642525 9051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:47.642549 9052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:47.642657 9046 server_base.cc:1034] running on GCE node
W20260927 21:12:47.642547 9054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:47.642928 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:47.643209 9046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:47.644368 9046 hybrid_clock.cc:648] HybridClock initialized: now 1790543567644349 us; error 30 us; skew 500 ppm
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:47.646816 9046 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:12:47.647225 9046 webserver.cc:460] Webserver started at http://127.0.98.62:35689/ using document root <none> and password file <none>
I20260927 21:12:47.647425 9046 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:47.647478 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:47.648779 9046 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:47.649401 9061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.649614 9046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:12:47.649689 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb"
format_stamp: "Formatted at 2026-09-27 21:12:46 on dist-test-slave-r9x1"
server_key: "81311e71b21482e9e7b81fe7764db821"
server_key_iv: "d4927be1eed1ef0d0351c00510489a45"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.650055 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:47.657874 9046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:47.660025 9046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:47.660148 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:47.664127 9046 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40661
I20260927 21:12:47.664172 9113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40661 every 8 connection(s)
I20260927 21:12:47.664681 9046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:12:47.665973 9114 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:12:47.666560 9114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: Bootstrap starting.
I20260927 21:12:47.667035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 9046
I20260927 21:12:47.667290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41013
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40661
--webserver_port=44379
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:12:47.675055 9114 log.cc:824] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:12:47.676970 9114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:12:47.677307 9114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: Bootstrap complete.
I20260927 21:12:47.679448 9114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } }
I20260927 21:12:47.679685 9114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8f62e1f2c64b769d6ee58602eb7bfb, State: Initialized, Role: FOLLOWER
I20260927 21:12:47.679848 9114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } }
I20260927 21:12:47.679917 9114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:12:47.679962 9114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:12:47.680008 9114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:12:47.681031 9114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } }
I20260927 21:12:47.681123 9114 leader_election.cc:302] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d8f62e1f2c64b769d6ee58602eb7bfb; no voters:
I20260927 21:12:47.681325 9114 leader_election.cc:288] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:12:47.681363 9118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:12:47.681465 9118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [term 2 LEADER]: Becoming Leader. State: Replica: 3d8f62e1f2c64b769d6ee58602eb7bfb, State: Running, Role: LEADER
I20260927 21:12:47.681556 9118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } }
I20260927 21:12:47.681595 9114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:12:47.681820 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } } }
I20260927 21:12:47.681888 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:47.681995 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d8f62e1f2c64b769d6ee58602eb7bfb. Latest consensus state: current_term: 2 leader_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8f62e1f2c64b769d6ee58602eb7bfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40661 } } }
I20260927 21:12:47.682050 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb [sys.catalog]: This master's current role is: LEADER
I20260927 21:12:47.682317 9122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:12:47.682816 9122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:12:47.682955 9122 catalog_manager.cc:1259] Loaded cluster ID: 3362c42d445f461eba43b243661c74c5
I20260927 21:12:47.682996 9122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:12:47.684142 9122 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:12:47.684324 9122 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3d8f62e1f2c64b769d6ee58602eb7bfb: Loaded TSK: 0
I20260927 21:12:47.684650 9122 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:12:47.750701 9116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:47.750933 9116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:47.750968 9116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:47.752310 9116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:47.752360 9116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:47.752403 9116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:12:47.754144 9116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44379
--tserver_master_addrs=127.0.98.62:40661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:47.754417 9116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:47.754647 9116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:47.759932 9143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:47.759970 9141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:47.759927 9140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:47.760298 9116 server_base.cc:1034] running on GCE node
I20260927 21:12:47.760545 9116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:47.760802 9116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:47.761977 9116 hybrid_clock.cc:648] HybridClock initialized: now 1790543567761961 us; error 32 us; skew 500 ppm
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543567, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:47.764643 9116 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 21:12:47.765247 9116 webserver.cc:460] Webserver started at http://127.0.98.1:44379/ using document root <none> and password file <none>
I20260927 21:12:47.765559 9116 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:47.765642 9116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:47.767531 9116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:12:47.768353 9150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.768564 9116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:12:47.768654 9116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "5c72349a9b8b45a2aa789c1a9471c9f0"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "66fda849da4dfafd1d2d6f6e8068ef56"
server_key_iv: "eaa8095931b530b62f4222aca37531ad"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.769049 9116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:47.795145 9116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:47.797816 9116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:47.797943 9116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:47.798194 9116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:47.798499 9116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:47.798533 9116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.798564 9116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:47.798588 9116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.804466 9116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41013
I20260927 21:12:47.804513 9263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41013 every 8 connection(s)
I20260927 21:12:47.804927 9116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:12:47.812903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 9116
I20260927 21:12:47.813349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41173
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40661
--webserver_port=40651
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:47.818607 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.805696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:36075)
0927 21:12:47.805881 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:47.805944 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:47.806612 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:47.809942 (+ 3330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:47.809947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:47.810721 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:47.811023 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.811031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.812318 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.812319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.813709 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.813714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.814641 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.814642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.815018 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.815734 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:47.815746 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:47.818339 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
W20260927 21:12:47.818786 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:47.805790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36075 (local address 127.0.98.62:40661)
0927 21:12:47.809456 (+ 3666us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:47.809466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:47.809503 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:12:47.809621 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:47.809627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:47.809677 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:47.809821 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:47.811206 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.812164 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.813843 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.814517 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.814652 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.818577 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36075: BlockingRecv error: recv got EOF from 127.0.98.1:36075 (error 108)
Metrics: {"server-negotiator.queue_time_us":3596,"thread_start_us":75,"threads_started":1}
W20260927 21:12:47.818869 9264 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40661 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40661: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:47.823995 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.819100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:54639)
0927 21:12:47.819158 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:47.819168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:47.819207 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:47.819352 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:47.819352 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:47.819477 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:47.819528 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.819529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.820442 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.820443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.820969 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.820970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.821476 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.821477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.821528 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.821939 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:47.821944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:47.823937 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 21:12:47.824136 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:47.819161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54639 (local address 127.0.98.62:40661)
0927 21:12:47.819217 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:47.819219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:47.819228 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:47.819260 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:47.819261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:47.819275 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:47.819340 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:47.819589 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.820343 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.821051 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.821401 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.821483 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.824077 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54639: BlockingRecv error: recv got EOF from 127.0.98.1:54639 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:47.828434 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.824272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:53429)
0927 21:12:47.824317 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:47.824327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:47.824363 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:47.824483 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:47.824484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:47.824601 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:47.824636 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.824637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.825251 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.825252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.825780 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.825782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.826109 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.826110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.826159 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.826565 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:47.826570 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:47.828362 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 21:12:47.828539 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:47.824334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53429 (local address 127.0.98.62:40661)
0927 21:12:47.824370 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:47.824372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:47.824391 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:12:47.824417 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:47.824418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:47.824428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:47.824474 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:47.824700 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.825180 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.825863 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.826050 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.826108 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.828486 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53429: BlockingRecv error: recv got EOF from 127.0.98.1:53429 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 21:12:47.828599 9264 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:12:47.898098 9268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:47.898335 9268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:47.898361 9268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:47.899667 9268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:47.899719 9268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:47.899748 9268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:12:47.901304 9268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40651
--tserver_master_addrs=127.0.98.62:40661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:47.901556 9268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:47.901780 9268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:47.907025 9276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:47.907004 9273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:47.907210 9268 server_base.cc:1034] running on GCE node
W20260927 21:12:47.907001 9274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:47.907476 9268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:47.907725 9268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:47.908902 9268 hybrid_clock.cc:648] HybridClock initialized: now 1790543567908894 us; error 37 us; skew 500 ppm
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543567, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:47.917018 9268 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 21:12:47.917551 9268 webserver.cc:460] Webserver started at http://127.0.98.2:40651/ using document root <none> and password file <none>
I20260927 21:12:47.917783 9268 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:47.917845 9268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:47.919351 9268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:47.919986 9283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.920140 9268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:12:47.920221 9268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "5f487c2318f3407d8a519408aebde94e"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "c9f7a4d9794a8e0c6585687d112574e1"
server_key_iv: "95f89b8685d3e68233d7a04343e1992e"
server_key_version: "encryptionkey@0"
I20260927 21:12:47.920569 9268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:47.940642 9268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:47.942797 9268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:47.942965 9268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:47.943187 9268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:47.943488 9268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:47.943527 9268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.943557 9268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:47.943580 9268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:47.949102 9268 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41173
I20260927 21:12:47.949149 9396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41173 every 8 connection(s)
I20260927 21:12:47.949554 9268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:12:47.949756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 9268
I20260927 21:12:47.950019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37235
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40661
--webserver_port=43665
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:12:47.958523 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:47.950344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40825 (local address 127.0.98.62:40661)
0927 21:12:47.950397 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:47.950399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:47.950683 (+ 284us) server_negotiation.cc:407] Connection header received
0927 21:12:47.951186 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:47.951187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:47.951203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:47.951277 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:47.952562 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.953376 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.954525 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.954850 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.954941 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.958453 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40825: BlockingRecv error: recv got EOF from 127.0.98.2:40825 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:12:47.958549 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.950375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:40825)
0927 21:12:47.950545 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:47.950590 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:47.951075 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:47.951366 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:47.951371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:47.952186 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:47.952444 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.952450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.953468 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.953469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.954365 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.954367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.954969 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.954970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.955095 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.955787 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:47.955796 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:47.958314 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
W20260927 21:12:47.958878 9397 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40661 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40661: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:47.963515 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.959186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:60023)
0927 21:12:47.959256 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:47.959266 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:47.959308 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:47.959431 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:47.959432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:47.959571 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:47.959618 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.959620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.960230 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.960231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.960738 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.960740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.961068 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.961069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.961139 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.961629 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:47.961634 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:47.963437 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:12:47.963596 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:47.959256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60023 (local address 127.0.98.62:40661)
0927 21:12:47.959323 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:47.959325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:47.959333 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:47.959361 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:47.959362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:47.959373 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:47.959418 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:47.959686 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.960159 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.960824 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.961006 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.961062 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.963545 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60023: BlockingRecv error: recv got EOF from 127.0.98.2:60023 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:47 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:47.968657 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:47.963890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:37987)
0927 21:12:47.963946 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:47.963958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:47.964007 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:47.964135 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:47.964136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:47.964296 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:47.964337 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.964339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.965019 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.965020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.965887 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:47.965891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.966258 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.966258 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:47.966312 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.966733 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:47.966739 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:47.968582 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 21:12:47.968812 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:47.963952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37987 (local address 127.0.98.62:40661)
0927 21:12:47.963998 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:47.964000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:47.964008 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:47.964066 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:47.964067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:47.964075 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:47.964118 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:47.964436 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.964942 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.965988 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:47.966184 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:47.966249 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:47.968677 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37987: BlockingRecv error: recv got EOF from 127.0.98.2:37987 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 21:12:47.968833 9397 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:12:48.033967 9400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:12:48.034189 9400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:12:48.034214 9400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:12:48.035425 9400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:12:48.035466 9400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:12:48.035494 9400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:12:48.036954 9400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43665
--tserver_master_addrs=127.0.98.62:40661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:12:48.037160 9400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:12:48.037367 9400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:12:48.042208 9406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:48.042237 9405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:12:48.042248 9408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:12:48.042397 9400 server_base.cc:1034] running on GCE node
I20260927 21:12:48.042691 9400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:12:48.042934 9400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:12:48.044093 9400 hybrid_clock.cc:648] HybridClock initialized: now 1790543568044077 us; error 31 us; skew 500 ppm
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543568, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:12:48.046674 9400 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 21:12:48.047116 9400 webserver.cc:460] Webserver started at http://127.0.98.3:43665/ using document root <none> and password file <none>
I20260927 21:12:48.047349 9400 fs_manager.cc:359] Metadata directory not provided
I20260927 21:12:48.047415 9400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:12:48.048867 9400 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:48.049541 9415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:48.049729 9400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:12:48.049810 9400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "e61be972eac341a28117e967f77073d9"
format_stamp: "Formatted at 2026-09-27 21:12:47 on dist-test-slave-r9x1"
server_key: "11b7eaa85b124b6496e95b30d73d1386"
server_key_iv: "479780c85b9dd4b90c8ad1c5f899d747"
server_key_version: "encryptionkey@0"
I20260927 21:12:48.050166 9400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:12:48.059439 9400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:12:48.061700 9400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:12:48.061849 9400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:12:48.062088 9400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:12:48.062413 9400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:12:48.062449 9400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:48.062479 9400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:12:48.062501 9400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:12:48.068022 9400 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37235
I20260927 21:12:48.068082 9528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37235 every 8 connection(s)
I20260927 21:12:48.068427 9400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:48.075598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 9400
W20260927 21:12:48.076217 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:48.069113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39119 (local address 127.0.98.62:40661)
0927 21:12:48.069179 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:48.069189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:48.069452 (+ 263us) server_negotiation.cc:407] Connection header received
0927 21:12:48.069982 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:48.069983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:48.069996 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:48.070050 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:48.071040 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.071683 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.072828 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.073028 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.073084 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.076162 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39119: BlockingRecv error: recv got EOF from 127.0.98.3:39119 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:12:48.076284 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:48.069109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:39119)
0927 21:12:48.069328 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:48.069367 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:48.069880 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:48.070127 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:48.070130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:48.070728 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:48.070937 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.070942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.071811 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.071812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.072740 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.072743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.073103 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.073104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.073219 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.073702 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:48.073711 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:48.076074 (+ 2363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
W20260927 21:12:48.076502 9529 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40661 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40661: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:48.082157 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:48.076700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:38325)
0927 21:12:48.076776 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:48.076785 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:48.076829 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:48.076955 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:48.076956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:48.077090 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:48.077155 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.077157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.078121 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.078122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.078981 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.078984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.079456 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.079457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.079528 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.079990 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:48.079995 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:48.082082 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 21:12:48.082230 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:48.076743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38325 (local address 127.0.98.62:40661)
0927 21:12:48.076788 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:48.076790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:48.076817 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:12:48.076884 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:48.076884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:48.076891 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:48.076933 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:48.077223 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.078032 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.079089 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.079336 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.079377 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.082162 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38325: BlockingRecv error: recv got EOF from 127.0.98.3:38325 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":16}
I20260927 21:12:48.084218 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:48.076135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.0.98.62:40661)
0927 21:12:48.076237 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:48.076238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:48.076248 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:48.076473 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:48.076474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:48.076483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:48.076537 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:48.077244 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.078032 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.079117 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.079476 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.079542 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.080912 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:12:48.080919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:12:48.080919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:12:48.080934 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:12:48.082927 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:48.083238 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:48.083242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:48.083242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:48.083335 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:12:48.083514 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:12:48.083515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:12:48.083516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:12:48.083754 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:12:48.083890 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 21:12:48.084008 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:12:48.084118 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:48.086959 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:48.082546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:39093)
0927 21:12:48.082657 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:48.082669 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:48.082742 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:48.082855 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:48.082856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:48.082995 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:48.083027 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.083029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.083644 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.083645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.084294 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.084307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.084690 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.084691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.084749 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.085127 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:48.085132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:48.086905 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 21:12:48.087075 9267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:48.082617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39093 (local address 127.0.98.62:40661)
0927 21:12:48.082667 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:48.082670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:48.082724 (+ 54us) server_negotiation.cc:407] Connection header received
0927 21:12:48.082798 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:48.082799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:48.082808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:48.082852 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:48.083090 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.083575 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.084381 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.084636 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.084683 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.087016 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39093: BlockingRecv error: recv got EOF from 127.0.98.3:39093 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 21:12:48.087144 9529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:48.833886 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:48.829074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:38819)
0927 21:12:48.829194 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:48.829204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:48.829291 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:48.829431 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:48.829432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:48.829600 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:48.829648 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.829649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.830318 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.830318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.830884 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.830885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.831260 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.831261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.831307 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.831819 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:48.831823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:48.833800 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
W20260927 21:12:48.833995 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:48.829120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38819 (local address 127.0.98.62:40661)
0927 21:12:48.829245 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:48.829246 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:48.829255 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:48.829350 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:48.829351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:48.829359 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:48.829419 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:48.829735 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.830246 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.830977 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.831182 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.831243 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.833919 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38819: BlockingRecv error: recv got EOF from 127.0.98.1:38819 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:48 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:48.973953 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:48.969288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:58041)
0927 21:12:48.969424 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:48.969437 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:48.969524 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:48.969659 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:48.969660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:48.969831 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:48.969880 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.969882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.970603 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.970604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.971135 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:48.971137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.971480 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.971481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:48.971530 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.971997 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:48.972001 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:48.973861 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
W20260927 21:12:48.974035 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:48.969407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58041 (local address 127.0.98.62:40661)
0927 21:12:48.969467 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:48.969469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:48.969502 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:12:48.969585 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:48.969586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:48.969597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:48.969645 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:48.969955 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.970520 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.971214 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:48.971417 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:48.971474 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:48.973984 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58041: BlockingRecv error: recv got EOF from 127.0.98.2:58041 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 21:12:49 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:49 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:49.092636 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:49.087625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:55581)
0927 21:12:49.087803 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:49.087817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:49.087947 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:49.088130 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:49.088131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:49.088332 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:49.088384 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:49.088385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.089030 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.089031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:49.089670 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:49.089673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.090048 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.090049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:49.090103 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:49.090674 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:49.090679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:49.092566 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
W20260927 21:12:49.092726 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:49.087749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55581 (local address 127.0.98.62:40661)
0927 21:12:49.087844 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:49.087846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:49.087883 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:12:49.088051 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:49.088052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:49.088065 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:49.088113 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:49.088447 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.088951 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.089761 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.089959 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.090018 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:49.092664 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55581: BlockingRecv error: recv got EOF from 127.0.98.3:55581 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 21:12:49 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:49 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:49.839020 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:49.834379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:48411)
0927 21:12:49.834532 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:49.834544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:49.834615 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:49.834778 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:49.834780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:49.834957 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:49.835007 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:49.835009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.835723 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.835724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:49.836277 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:49.836279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.836640 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.836640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:49.836677 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:49.837179 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:49.837185 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:49.838951 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
W20260927 21:12:49.839118 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:49.834486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48411 (local address 127.0.98.62:40661)
0927 21:12:49.834603 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:49.834604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:49.834614 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:49.834663 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:49.834663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:49.834673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:49.834733 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:49.835105 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.835645 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.836370 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.836579 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.836636 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:49.839044 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48411: BlockingRecv error: recv got EOF from 127.0.98.1:48411 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
Sep 27 21:12:49 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:49 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:49.978938 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:49.974457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:37277)
0927 21:12:49.974594 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:49.974605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:49.974658 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:49.974810 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:49.974811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:49.974991 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:49.975039 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:49.975041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.975688 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.975689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:49.976217 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:49.976218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.976559 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.976560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:49.976613 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:49.977094 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:49.977100 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:49.978853 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
W20260927 21:12:49.979020 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:49.974635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37277 (local address 127.0.98.62:40661)
0927 21:12:49.974700 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:49.974702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:49.974710 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:49.974732 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:49.974733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:49.974745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:49.974787 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:49.975114 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.975610 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.976305 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:49.976494 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:49.976553 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:49.978967 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37277: BlockingRecv error: recv got EOF from 127.0.98.2:37277 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 21:12:50 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:50 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:50.098161 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:50.093219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:44761)
0927 21:12:50.093379 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:50.093409 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:50.093461 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:50.093689 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:50.093690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:50.093850 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:50.093899 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:50.093901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.094681 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.094682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:50.095373 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:50.095376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.095779 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.095780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:50.095842 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:50.096341 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:50.096345 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:50.098100 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
W20260927 21:12:50.098261 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:50.093444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44761 (local address 127.0.98.62:40661)
0927 21:12:50.093503 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:50.093505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:50.093515 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:50.093545 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:50.093546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:50.093557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:50.093635 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:50.093978 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.094573 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.095471 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.095684 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.095748 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:50.098200 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44761: BlockingRecv error: recv got EOF from 127.0.98.3:44761 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 21:12:50 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:50 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:50.844086 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:50.839539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:50263)
0927 21:12:50.839667 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:50.839677 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:50.839731 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:50.839845 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:50.839845 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:50.839997 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:50.840049 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:50.840051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.840746 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.840747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:50.841346 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:50.841349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.841761 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.841762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:50.841807 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:50.842306 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:50.842311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:50.844009 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
W20260927 21:12:50.844188 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:50.839625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50263 (local address 127.0.98.62:40661)
0927 21:12:50.839738 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:50.839740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:50.839748 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:12:50.839771 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:50.839771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:50.839781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:50.839835 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:50.840109 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.840642 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.841466 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.841673 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.841733 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:50.844126 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50263: BlockingRecv error: recv got EOF from 127.0.98.1:50263 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
Sep 27 21:12:50 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:50 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:50.984441 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:50.979463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:38461)
0927 21:12:50.979632 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:50.979650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:50.979871 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:50.980068 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:50.980069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:50.980232 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:50.980286 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:50.980287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.981021 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.981022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:50.981703 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:50.981705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.982023 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.982024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:50.982062 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:50.982600 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:50.982605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:50.984378 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
W20260927 21:12:50.984622 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:50.979585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38461 (local address 127.0.98.62:40661)
0927 21:12:50.979633 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:50.979635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:50.979901 (+ 266us) server_negotiation.cc:407] Connection header received
0927 21:12:50.979945 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:50.979946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:50.979966 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:50.980046 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:50.980392 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.980920 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.981765 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:50.981966 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:50.982021 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:50.984568 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38461: BlockingRecv error: recv got EOF from 127.0.98.2:38461 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 21:12:51 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:51 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:51.104094 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:51.098796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:53035)
0927 21:12:51.098986 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:51.098999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:51.099065 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:51.099202 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:51.099203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:51.099365 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:51.099418 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:51.099421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.100160 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.100162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:51.100709 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:51.100711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.101131 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.101132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:51.101172 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:51.101785 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:51.101791 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:51.104012 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
W20260927 21:12:51.104233 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:51.098960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53035 (local address 127.0.98.62:40661)
0927 21:12:51.099034 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:51.099037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:51.099051 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:12:51.099117 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:51.099118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:51.099131 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:51.099181 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:51.099530 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.100051 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.100813 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.101055 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.101129 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:51.104168 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53035: BlockingRecv error: recv got EOF from 127.0.98.3:53035 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 21:12:51 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:51 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:51.849153 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:51.844672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:58991)
0927 21:12:51.844816 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:51.844827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:51.844882 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:51.845022 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:51.845023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:51.845182 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:51.845231 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:51.845233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.845959 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.845960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:51.846556 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:51.846560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.846936 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.846937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:51.846984 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:51.847434 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:51.847439 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:51.849091 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
W20260927 21:12:51.849337 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:51.844769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58991 (local address 127.0.98.62:40661)
0927 21:12:51.844888 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:51.844890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:51.844901 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:12:51.844927 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:51.844928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:51.844938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:51.845011 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:51.845333 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.845876 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.846625 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.846863 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.846926 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:51.849232 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58991: BlockingRecv error: recv got EOF from 127.0.98.1:58991 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
Sep 27 21:12:51 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:51 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:51.989509 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:51.985000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:47969)
0927 21:12:51.985124 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:51.985134 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:51.985194 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:51.985361 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:51.985361 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:51.985512 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:51.985562 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:51.985563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.986190 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.986191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:51.986737 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:51.986739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.987081 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.987082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:51.987144 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:51.987590 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:51.987595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:51.989434 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
W20260927 21:12:51.989593 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:51.985103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47969 (local address 127.0.98.62:40661)
0927 21:12:51.985154 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:51.985156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:51.985179 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:12:51.985244 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:51.985245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:51.985256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:51.985347 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:51.985618 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.986106 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.986811 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:51.986994 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:51.987044 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:51.989532 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47969: BlockingRecv error: recv got EOF from 127.0.98.2:47969 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 21:12:52 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:52 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:52.110003 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:52.104769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:58501)
0927 21:12:52.104938 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:52.104953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:52.105022 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:52.105424 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:52.105425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:52.105708 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:52.105769 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:52.105771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.106579 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.106580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:52.107122 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:52.107124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.107519 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.107520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:52.107578 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:52.108061 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:52.108066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:52.109941 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
W20260927 21:12:52.110154 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:52.104972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58501 (local address 127.0.98.62:40661)
0927 21:12:52.105141 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:52.105143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:52.105161 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:12:52.105234 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:52.105236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:52.105258 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:52.105390 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:52.105923 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.106475 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.107230 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.107424 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.107489 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:52.110091 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58501: BlockingRecv error: recv got EOF from 127.0.98.3:58501 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 27 21:12:52 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:52 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:52.854494 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:52.849678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:42107)
0927 21:12:52.849809 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:52.849819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:52.849875 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:52.850023 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:52.850023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:52.850178 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:52.850226 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:52.850228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.850894 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.850895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:52.851428 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:52.851431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.851825 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.851826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:52.851869 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:52.852374 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:52.852379 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:52.854432 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
W20260927 21:12:52.854610 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:52.849772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42107 (local address 127.0.98.62:40661)
0927 21:12:52.849883 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:52.849885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:52.849919 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:12:52.849942 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:52.849942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:52.849959 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:52.850010 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:52.850293 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.850792 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.851540 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.851758 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.851820 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:52.854537 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42107: BlockingRecv error: recv got EOF from 127.0.98.1:42107 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
Sep 27 21:12:52 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:52 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:52.998246 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:52.989997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:46075)
0927 21:12:52.990134 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:52.990147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:52.990216 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:52.990338 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:52.990339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:52.990518 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:52.990572 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:52.990574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.991223 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.991224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:52.991773 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:52.991774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.992120 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.992121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:52.992174 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:52.996151 (+ 3977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:52.996157 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:52.998173 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
W20260927 21:12:52.998347 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:52.990091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46075 (local address 127.0.98.62:40661)
0927 21:12:52.990150 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:52.990152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:52.990198 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:12:52.990268 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:52.990269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:52.990281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:52.990328 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:52.990648 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.991145 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.991857 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:52.992056 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:52.992112 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:52.998290 (+ 6178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46075: BlockingRecv error: recv got EOF from 127.0.98.2:46075 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 21:12:53 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:53 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:53.115077 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:53.110563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:41933)
0927 21:12:53.110702 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:53.110713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:53.110793 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:53.110962 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:53.110963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:53.111127 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:53.111176 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:53.111178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:53.111839 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:53.111841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:53.112401 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:53.112403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:53.112765 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:53.112766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:53.112812 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:53.113318 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:53.113323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:53.114995 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
W20260927 21:12:53.115175 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:53.110602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41933 (local address 127.0.98.62:40661)
0927 21:12:53.110647 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:53.110649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:53.110791 (+ 142us) server_negotiation.cc:407] Connection header received
0927 21:12:53.110840 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:53.110841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:53.110856 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:53.110908 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:53.111251 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:53.111747 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:53.112496 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:53.112695 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:53.112756 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:53.115104 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41933: BlockingRecv error: recv got EOF from 127.0.98.3:41933 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 21:12:53 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:53 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:53.859644 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:53.855030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:45989)
0927 21:12:53.855171 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:53.855185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:53.855238 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:53.855385 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:53.855385 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:53.855548 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:53.855683 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:53.855690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:53.856414 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:53.856415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:53.856961 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:53.856962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:53.857316 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:53.857317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:53.857368 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:53.857792 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:53.857796 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:53.859541 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
W20260927 21:12:53.859706 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:53.855116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45989 (local address 127.0.98.62:40661)
0927 21:12:53.855239 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:53.855241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:53.855281 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:12:53.855307 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:53.855308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:53.855318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:53.855370 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:53.855816 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:53.856344 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:53.857024 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:53.857227 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:53.857295 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:53.859646 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45989: BlockingRecv error: recv got EOF from 127.0.98.1:45989 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
Sep 27 21:12:53 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:53 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:54.003860 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:53.998894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:39645)
0927 21:12:53.999042 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:53.999055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:53.999134 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:53.999331 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:53.999332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:53.999593 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:53.999645 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:53.999647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.000306 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.000308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:54.000898 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:54.000899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.001424 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.001425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:54.001482 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:54.001980 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:54.001985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:54.003718 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
W20260927 21:12:54.003924 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:53.999001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39645 (local address 127.0.98.62:40661)
0927 21:12:53.999067 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:53.999069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:53.999109 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:12:53.999226 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:53.999227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:53.999239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:53.999312 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:53.999715 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.000208 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.001029 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.001236 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.001326 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:54.003858 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39645: BlockingRecv error: recv got EOF from 127.0.98.2:39645 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 21:12:54 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:54 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:54.120422 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:54.115671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:55289)
0927 21:12:54.115903 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:54.115915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:54.115994 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:54.116137 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:54.116138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:54.116309 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:54.116356 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:54.116358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.117008 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.117009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:54.117588 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:54.117591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.117913 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.117914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:54.117974 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:54.118453 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:54.118459 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:54.120343 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":155,"threads_started":1}
W20260927 21:12:54.120496 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:54.115789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55289 (local address 127.0.98.62:40661)
0927 21:12:54.115875 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:54.115877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:54.115990 (+ 113us) server_negotiation.cc:407] Connection header received
0927 21:12:54.116058 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:54.116059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:54.116073 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:54.116119 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:54.116430 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.116916 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.117671 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.117852 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.117907 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:54.120431 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55289: BlockingRecv error: recv got EOF from 127.0.98.3:55289 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 21:12:54 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:54 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:54.864848 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:54.860167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:37389)
0927 21:12:54.860309 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:54.860321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:54.860384 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:54.860509 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:54.860510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:54.860664 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:54.860711 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:54.860712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.861360 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.861361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:54.861905 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:54.861907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.862313 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.862314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:54.862372 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:54.862845 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:54.862849 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:54.864784 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
W20260927 21:12:54.865015 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:54.860251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37389 (local address 127.0.98.62:40661)
0927 21:12:54.860375 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:54.860377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:54.860386 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:54.860430 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:54.860430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:54.860441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:54.860500 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:54.860765 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.861289 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.862004 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:54.862217 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:54.862280 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:54.864943 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37389: BlockingRecv error: recv got EOF from 127.0.98.1:37389 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
Sep 27 21:12:55 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:55 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:55.008991 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:55.004361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:52115)
0927 21:12:55.004535 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:55.004545 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:55.004644 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:55.004784 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:55.004785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:55.004953 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:55.005000 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:55.005001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.005697 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.005698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:55.006318 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:55.006321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.006676 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.006684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:55.006739 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:55.007215 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:55.007220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:55.008903 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
W20260927 21:12:55.009075 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:55.004422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52115 (local address 127.0.98.62:40661)
0927 21:12:55.004469 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:55.004483 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:55.004646 (+ 163us) server_negotiation.cc:407] Connection header received
0927 21:12:55.004703 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:55.004704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:55.004717 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:55.004767 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:55.005082 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.005622 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.006425 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.006614 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.006670 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:55.009022 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52115: BlockingRecv error: recv got EOF from 127.0.98.2:52115 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 21:12:55 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:55 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:55.125370 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:55.120865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:37961)
0927 21:12:55.120985 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:55.120996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:55.121047 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:55.121181 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:55.121182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:55.121356 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:55.121406 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:55.121408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.122076 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.122077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:55.122681 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:55.122684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.123010 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.123011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:55.123051 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:55.123531 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:55.123536 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:55.125301 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
W20260927 21:12:55.125464 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:55.120973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37961 (local address 127.0.98.62:40661)
0927 21:12:55.121027 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:55.121029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:55.121040 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:12:55.121095 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:55.121096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:55.121108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:55.121171 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:55.121465 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.121977 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.122763 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.122952 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.123006 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:55.125396 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37961: BlockingRecv error: recv got EOF from 127.0.98.3:37961 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 21:12:55 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:55 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:55.869930 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:55.865354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:37323)
0927 21:12:55.865506 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:55.865516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:55.865571 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:55.865732 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:55.865733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:55.865893 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:55.865942 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:55.865944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.866682 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.866683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:55.867242 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:55.867244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.867599 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.867600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:55.867635 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:55.868135 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:55.868139 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:55.869867 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
W20260927 21:12:55.870085 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:55.865453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37323 (local address 127.0.98.62:40661)
0927 21:12:55.865565 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:55.865566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:55.865575 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:55.865612 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:55.865613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:55.865622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:55.865691 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:55.866059 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.866577 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.867337 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:55.867536 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:55.867598 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:55.870006 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37323: BlockingRecv error: recv got EOF from 127.0.98.1:37323 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
Sep 27 21:12:56 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:56 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:56.014019 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:56.009543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:48841)
0927 21:12:56.009659 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:56.009670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:56.009762 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:56.009894 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:56.009894 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:56.010076 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:56.010121 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:56.010123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.010915 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.010916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:56.011423 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:56.011425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.011752 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.011753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:56.011803 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:56.012244 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:56.012248 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:56.013941 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
W20260927 21:12:56.014103 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:56.009656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48841 (local address 127.0.98.62:40661)
0927 21:12:56.009740 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:56.009742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:56.009753 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:12:56.009830 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:56.009831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:56.009843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:56.009890 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:56.010196 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.010836 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.011503 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.011686 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.011742 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:56.014051 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48841: BlockingRecv error: recv got EOF from 127.0.98.2:48841 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 21:12:56 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:56 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:56.130416 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:56.125862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:46167)
0927 21:12:56.125991 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:56.126005 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:56.126082 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:56.126224 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:56.126225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:56.126393 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:56.126437 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:56.126439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.127084 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.127085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:56.127607 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:56.127609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.127923 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.127924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:56.127976 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:56.128472 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:56.128478 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:56.130337 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
W20260927 21:12:56.130496 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:56.125964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46167 (local address 127.0.98.62:40661)
0927 21:12:56.126060 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:56.126063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:56.126074 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:12:56.126147 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:56.126148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:56.126161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:56.126210 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:56.126521 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.127009 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.127683 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.127863 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.127918 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:56.130443 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46167: BlockingRecv error: recv got EOF from 127.0.98.3:46167 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 21:12:56 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:56 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:56.875131 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:56.870423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:55939)
0927 21:12:56.870588 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:56.870604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:56.870677 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:56.870805 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:56.870806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:56.870964 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:56.871027 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:56.871028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.871774 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.871775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:56.872320 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:56.872322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.872702 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.872703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:56.872756 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:56.873213 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:56.873217 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:56.875050 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
W20260927 21:12:56.875214 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:56.870556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55939 (local address 127.0.98.62:40661)
0927 21:12:56.870686 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:56.870688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:56.870697 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:56.870723 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:56.870724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:56.870734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:56.870793 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:56.871089 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.871673 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.872400 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:56.872628 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:56.872696 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:56.875154 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55939: BlockingRecv error: recv got EOF from 127.0.98.1:55939 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
Sep 27 21:12:57 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:57 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:57.019295 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:57.014585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:49899)
0927 21:12:57.014758 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:57.014773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:57.014865 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:57.015015 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:57.015016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:57.015183 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:57.015230 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:57.015231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.015891 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.015892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:57.016481 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:57.016483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.016882 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.016883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:57.016946 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:57.017396 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:57.017400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:57.019219 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
W20260927 21:12:57.019374 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:57.014716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49899 (local address 127.0.98.62:40661)
0927 21:12:57.014813 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:57.014815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:57.014848 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:12:57.014944 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:57.014945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:57.014956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:57.015004 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:57.015309 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.015812 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.016576 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.016806 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.016880 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:57.019325 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49899: BlockingRecv error: recv got EOF from 127.0.98.2:49899 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 21:12:57 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:57 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:57.136222 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:57.130953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:40097)
0927 21:12:57.131099 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:57.131110 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:57.131177 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:57.131305 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:57.131306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:57.131476 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:57.131528 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:57.131530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.132259 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.132260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:57.133101 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:57.133107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.133541 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.133542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:57.133608 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:57.134092 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:57.134098 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:57.136132 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
W20260927 21:12:57.136255 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:57.131094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40097 (local address 127.0.98.62:40661)
0927 21:12:57.131151 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:57.131153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:57.131165 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:12:57.131230 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:57.131231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:57.131242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:57.131293 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:57.131603 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.132161 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.133195 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.133462 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.133520 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:57.136205 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40097: BlockingRecv error: recv got EOF from 127.0.98.3:40097 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 21:12:57 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:57 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:57.880738 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:57.875745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:35219)
0927 21:12:57.875885 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:57.875895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:57.875949 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:57.876205 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:57.876207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:57.876409 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:57.876467 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:57.876469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.877172 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.877173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:57.877825 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:57.877829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.878232 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.878233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:57.878279 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:57.878764 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:57.878768 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:57.880674 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
W20260927 21:12:57.880846 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:57.875868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35219 (local address 127.0.98.62:40661)
0927 21:12:57.876015 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:57.876017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:57.876033 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:12:57.876060 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:57.876061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:57.876071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:57.876147 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:57.876564 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.877096 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.877945 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:57.878158 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:57.878221 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:57.880773 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35219: BlockingRecv error: recv got EOF from 127.0.98.1:35219 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
Sep 27 21:12:58 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:58 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:58.027908 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:58.019938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:40941)
0927 21:12:58.020069 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:58.020084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:58.020146 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:58.020282 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:58.020283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:58.020438 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:58.020493 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:58.020495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.021145 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.021145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:58.021739 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:58.021742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.022123 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.022124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:58.022169 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:58.026020 (+ 3851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:58.026027 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:58.027845 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
W20260927 21:12:58.028059 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:58.020070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40941 (local address 127.0.98.62:40661)
0927 21:12:58.020123 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:58.020125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:58.020148 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:12:58.020213 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:58.020214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:58.020225 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:58.020266 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:58.020573 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.021083 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.021837 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.022049 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.022112 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:58.027973 (+ 5861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40941: BlockingRecv error: recv got EOF from 127.0.98.2:40941 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 21:12:58 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:58 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:58.141419 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:58.136766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:48309)
0927 21:12:58.136931 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:58.136944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:58.137034 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:58.137173 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:58.137174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:58.137399 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:58.137451 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:58.137452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.138115 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.138116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:58.138638 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:58.138639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.138969 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.138970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:58.139023 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:58.139539 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:58.139544 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:58.141340 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":59,"threads_started":1}
W20260927 21:12:58.141542 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:58.136911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48309 (local address 127.0.98.62:40661)
0927 21:12:58.136969 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:58.136971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:58.137012 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:12:58.137096 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:58.137097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:58.137112 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:58.137160 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:58.137538 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.138032 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.138722 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.138907 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.138966 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:58.141491 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48309: BlockingRecv error: recv got EOF from 127.0.98.3:48309 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 21:12:58 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:58 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:58.886108 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:58.881318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:57927)
0927 21:12:58.881475 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:58.881486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:58.881546 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:58.881734 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:58.881735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:58.881942 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:58.881992 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:58.881994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.882716 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.882717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:58.883449 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:58.883452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.883816 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.883816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:58.883854 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:58.884308 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:58.884312 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:58.886044 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
W20260927 21:12:58.886273 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:58.881423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57927 (local address 127.0.98.62:40661)
0927 21:12:58.881559 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:58.881561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:58.881572 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:12:58.881626 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:58.881627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:58.881640 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:58.881724 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:58.882081 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.882620 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.883531 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:58.883752 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:58.883813 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:58.886161 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57927: BlockingRecv error: recv got EOF from 127.0.98.1:57927 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
Sep 27 21:12:59 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:59 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:59.033383 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:59.028453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:32911)
0927 21:12:59.028576 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:59.028586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:59.028647 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:59.028765 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:59.028766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:59.028939 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:59.028988 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:59.028990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.029784 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.029785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:59.030363 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:59.030365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.030762 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.030763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:59.030814 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:59.031282 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:59.031286 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:59.033316 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
W20260927 21:12:59.033483 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:59.028571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32911 (local address 127.0.98.62:40661)
0927 21:12:59.028636 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:59.028637 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:59.028647 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:12:59.028695 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:59.028696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:59.028706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:59.028755 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:59.029080 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.029677 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.030471 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.030673 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.030733 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:59.033423 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32911: BlockingRecv error: recv got EOF from 127.0.98.2:32911 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 21:12:59 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:59 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:59.146624 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:59.141993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:49345)
0927 21:12:59.142143 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:59.142155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:59.142224 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:59.142356 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:59.142357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:59.142514 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:59.142561 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:59.142562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.143204 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.143205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:59.143752 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:59.143754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.144104 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.144105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:59.144149 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:59.144609 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:59.144614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:59.146564 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
W20260927 21:12:59.146723 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:59.142099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49345 (local address 127.0.98.62:40661)
0927 21:12:59.142168 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:59.142170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:59.142208 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:12:59.142280 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:59.142281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:59.142292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:59.142339 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:59.142615 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.143109 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.143856 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.144043 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.144094 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:59.146661 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49345: BlockingRecv error: recv got EOF from 127.0.98.3:49345 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 21:12:59 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:12:59 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:12:59.891510 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:12:59.886588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:42899)
0927 21:12:59.886719 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 21:12:59.886729 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:12:59.886786 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:12:59.887046 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:12:59.887047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:12:59.887224 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:12:59.887277 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:59.887279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.887981 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.887982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:59.888526 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:12:59.888528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.888896 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.888897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:12:59.888942 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:59.889536 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:12:59.889542 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:12:59.891445 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
W20260927 21:12:59.891641 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:12:59.886744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42899 (local address 127.0.98.62:40661)
0927 21:12:59.886880 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 21:12:59.886882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:12:59.886891 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:12:59.886917 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:12:59.886918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:12:59.886928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:12:59.887002 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:12:59.887382 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.887892 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.888616 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:12:59.888818 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:12:59.888874 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:12:59.891571 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42899: BlockingRecv error: recv got EOF from 127.0.98.1:42899 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
Sep 27 21:13:00 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:00 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:00.038447 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:00.033932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:55499)
0927 21:13:00.034088 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:00.034101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:00.034168 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:00.034346 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:00.034347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:00.034536 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:00.034582 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:00.034585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.035230 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.035231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:00.035760 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:00.035762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.036092 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.036093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:00.036145 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:00.036594 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:00.036598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:00.038359 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
W20260927 21:13:00.038533 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:00.034065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55499 (local address 127.0.98.62:40661)
0927 21:13:00.034150 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:00.034153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:00.034165 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:13:00.034230 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:00.034232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:00.034241 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:00.034333 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:00.034659 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.035151 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.035847 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.036032 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.036085 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:00.038481 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55499: BlockingRecv error: recv got EOF from 127.0.98.2:55499 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 21:13:00 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:00 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:00.151907 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:00.147198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:55211)
0927 21:13:00.147358 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:00.147368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:00.147431 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:00.147566 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:00.147566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:00.147772 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:00.147831 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:00.147834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.148537 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.148538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:00.149107 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:00.149109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.149509 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.149510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:00.149564 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:00.150011 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:00.150015 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:00.151846 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
W20260927 21:13:00.152000 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:00.147267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55211 (local address 127.0.98.62:40661)
0927 21:13:00.147324 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:00.147326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:00.147434 (+ 108us) server_negotiation.cc:407] Connection header received
0927 21:13:00.147486 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:00.147487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:00.147502 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:00.147559 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:00.147934 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.148445 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.149195 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.149430 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.149490 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:00.151939 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55211: BlockingRecv error: recv got EOF from 127.0.98.3:55211 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 21:13:00 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:00 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:00.902412 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:00.891940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:52925)
0927 21:13:00.892109 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:00.892148 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:00.892211 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:00.892388 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:00.892389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:00.892547 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:00.892596 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:00.892597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.893321 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.893322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:00.893876 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:00.893878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.894411 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.894412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:00.894460 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:00.900428 (+ 5968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:00.900440 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:00.902331 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
W20260927 21:13:00.902515 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:00.891981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52925 (local address 127.0.98.62:40661)
0927 21:13:00.892120 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:00.892122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:00.892205 (+ 83us) server_negotiation.cc:407] Connection header received
0927 21:13:00.892276 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:00.892277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:00.892287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:00.892337 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:00.892675 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.893187 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.893997 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:00.894219 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:00.894378 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:00.902440 (+ 8062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52925: BlockingRecv error: recv got EOF from 127.0.98.1:52925 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
Sep 27 21:13:01 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:01 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:01.043275 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:01.039021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:45737)
0927 21:13:01.039151 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:01.039161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:01.039218 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:01.039343 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:01.039344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:01.039514 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:01.039559 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:01.039561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.040198 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.040200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:01.040723 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:01.040725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.041051 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.041051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:01.041103 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:01.041587 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:01.041592 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:01.043200 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
W20260927 21:13:01.043360 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:01.039149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45737 (local address 127.0.98.62:40661)
0927 21:13:01.039222 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:01.039224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:01.039249 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:13:01.039280 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:01.039281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:01.039291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:01.039331 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:01.039635 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.040125 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.040798 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.040983 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.041044 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:01.043305 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45737: BlockingRecv error: recv got EOF from 127.0.98.2:45737 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 21:13:01 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:01 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:01.157133 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:01.152480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:51357)
0927 21:13:01.152611 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:01.152622 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:01.152704 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:01.152836 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:01.152837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:01.153032 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:01.153078 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:01.153081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.153945 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.153947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:01.154530 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:01.154533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.154949 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.154950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:01.155009 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:01.155401 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:01.155404 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:01.157042 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
W20260927 21:13:01.157209 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:01.152546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51357 (local address 127.0.98.62:40661)
0927 21:13:01.152612 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:01.152614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:01.152690 (+ 76us) server_negotiation.cc:407] Connection header received
0927 21:13:01.152761 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:01.152762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:01.152776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:01.152820 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:01.153156 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.153878 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.154604 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.154875 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.154935 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:01.157157 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51357: BlockingRecv error: recv got EOF from 127.0.98.3:51357 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 21:13:01 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:01 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:01.907956 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:01.902985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:46779)
0927 21:13:01.903169 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:01.903180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:01.903234 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:01.903378 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:01.903379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:01.903549 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:01.903597 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:01.903599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.904344 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.904345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:01.905069 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:01.905073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.905536 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.905537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:01.905595 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:01.906142 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:01.906147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:01.907876 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
W20260927 21:13:01.908061 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:01.903111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46779 (local address 127.0.98.62:40661)
0927 21:13:01.903238 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:01.903240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:01.903250 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:13:01.903280 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:01.903281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:01.903291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:01.903382 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:01.903721 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.904237 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.905174 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:01.905458 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:01.905519 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:01.908000 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46779: BlockingRecv error: recv got EOF from 127.0.98.1:46779 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
Sep 27 21:13:02 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:02 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:02.048936 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:02.043927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:42051)
0927 21:13:02.044059 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:02.044071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:02.044123 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:02.044302 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:02.044303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:02.044465 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:02.044512 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:02.044513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.045468 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.045469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:02.046018 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:02.046020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.046452 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.046453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:02.046502 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:02.046992 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:02.046997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:02.048825 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
W20260927 21:13:02.049000 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:02.044069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42051 (local address 127.0.98.62:40661)
0927 21:13:02.044169 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:02.044171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:02.044181 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:13:02.044210 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:02.044211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:02.044221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:02.044292 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:02.044638 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.045358 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.046129 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.046377 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.046439 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:02.048949 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42051: BlockingRecv error: recv got EOF from 127.0.98.2:42051 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 21:13:02 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:02 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:02.163021 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:02.157669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.3:33671)
0927 21:13:02.157794 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:02.157805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:02.157869 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:02.157989 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:02.157990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:02.158165 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:02.158213 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:02.158214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.158868 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.158869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:02.159638 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:02.159646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.160193 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.160195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:02.160258 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:02.161010 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:02.161017 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:02.162945 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
W20260927 21:13:02.163167 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:02.157707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33671 (local address 127.0.98.62:40661)
0927 21:13:02.157751 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:02.157753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:02.157863 (+ 110us) server_negotiation.cc:407] Connection header received
0927 21:13:02.157925 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:02.157926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:02.157940 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:02.157985 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:02.158286 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.158790 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.159858 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.160106 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.160186 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:02.163111 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33671: BlockingRecv error: recv got EOF from 127.0.98.3:33671 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 21:13:02 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:02 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:02.913120 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:02.908486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.1:49825)
0927 21:13:02.908612 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:02.908623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:02.908673 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:02.908841 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:02.908841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:02.909003 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:02.909053 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:02.909054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.909745 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.909745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:02.910283 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:02.910285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.910653 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.910654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:02.910693 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:02.911128 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:02.911132 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:02.913057 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
W20260927 21:13:02.913290 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:02.908549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49825 (local address 127.0.98.62:40661)
0927 21:13:02.908675 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:02.908677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:02.908703 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:13:02.908724 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:02.908725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:02.908736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:02.908832 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:02.909124 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.909673 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.910378 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:02.910585 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:02.910650 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:02.913182 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49825: BlockingRecv error: recv got EOF from 127.0.98.1:49825 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
Sep 27 21:13:03 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:13:03 dist-test-slave-r9x1 krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:13:03.059439 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.049590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40661 (local address 127.0.98.2:53003)
0927 21:13:03.049706 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:03.049718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:03.049777 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:03.049898 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:03.049898 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:03.050076 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:03.050121 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:03.050122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.050782 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.050783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:03.051289 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:03.051291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.051619 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.051620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:03.051673 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:03.057330 (+ 5657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:03.057337 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:03.059336 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40661: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
W20260927 21:13:03.059551 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:03.049690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53003 (local address 127.0.98.62:40661)
0927 21:13:03.049761 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:03.049763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:03.049772 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:13:03.049833 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:03.049834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:03.049847 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:03.049893 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:03.050194 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.050704 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.051370 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.051559 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.051615 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:03.059495 (+ 7880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53003: BlockingRecv error: recv got EOF from 127.0.98.2:53003 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:13:03.081851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 9116
I20260927 21:13:03.085943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 9268
I20260927 21:13:03.090137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 9400
I20260927 21:13:03.094197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 9046
2026-09-27T21:13:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17374 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T21:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:13:03Z Disabled control of system clock
I20260927 21:13:03.120725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39045
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:13:03.201097 9596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:03.201421 9596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:03.201452 9596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:03.202800 9596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:13:03.202848 9596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:03.202860 9596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:03.202872 9596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:13:03.202884 9596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:13:03.204298 9596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:03.204519 9596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:03.204808 9596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:03.210726 9602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.210737 9596 server_base.cc:1034] running on GCE node
W20260927 21:13:03.210743 9601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:03.210868 9604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.211225 9596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:03.211547 9596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:03.212720 9596 hybrid_clock.cc:648] HybridClock initialized: now 1790543583212708 us; error 30 us; skew 500 ppm
I20260927 21:13:03.214205 9596 webserver.cc:460] Webserver started at http://127.0.98.62:38599/ using document root <none> and password file <none>
I20260927 21:13:03.214434 9596 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:03.214488 9596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:03.214617 9596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:03.215883 9596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "17b91a14f5d44f4eb21c0368c39ea492"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "accca2b887edd608220cc6568c4210f0"
server_key_iv: "51256c9592c999343ba88f27a2dfea48"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.216261 9596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "17b91a14f5d44f4eb21c0368c39ea492"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "accca2b887edd608220cc6568c4210f0"
server_key_iv: "51256c9592c999343ba88f27a2dfea48"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.217638 9596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:13:03.218335 9610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.218492 9596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.218573 9596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "17b91a14f5d44f4eb21c0368c39ea492"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "accca2b887edd608220cc6568c4210f0"
server_key_iv: "51256c9592c999343ba88f27a2dfea48"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.218631 9596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:03.226629 9596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:03.226931 9596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:03.227038 9596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:03.231156 9596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39045
I20260927 21:13:03.231237 9662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39045 every 8 connection(s)
I20260927 21:13:03.231544 9596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:13:03.232221 9663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:03.234949 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492: Bootstrap starting.
I20260927 21:13:03.235423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 9596
I20260927 21:13:03.235507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:13:03.235590 9663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:03.235684 392 external_mini_cluster.cc:1428] Setting key 86e68892adc7fc220826ec7ca6683ada
I20260927 21:13:03.235888 9663 log.cc:824] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:03.236653 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492: No bootstrap required, opened a new log
I20260927 21:13:03.238907 9663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b91a14f5d44f4eb21c0368c39ea492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39045 } }
I20260927 21:13:03.239099 9663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:03.239130 9663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b91a14f5d44f4eb21c0368c39ea492, State: Initialized, Role: FOLLOWER
I20260927 21:13:03.239259 9663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b91a14f5d44f4eb21c0368c39ea492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39045 } }
I20260927 21:13:03.239360 9663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:13:03.239403 9663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:13:03.239455 9663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:03.240057 9663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b91a14f5d44f4eb21c0368c39ea492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39045 } }
I20260927 21:13:03.240172 9663 leader_election.cc:302] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17b91a14f5d44f4eb21c0368c39ea492; no voters:
I20260927 21:13:03.240350 9663 leader_election.cc:288] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:13:03.240398 9668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:13:03.240574 9668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [term 1 LEADER]: Becoming Leader. State: Replica: 17b91a14f5d44f4eb21c0368c39ea492, State: Running, Role: LEADER
I20260927 21:13:03.240629 9663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:13:03.240746 9668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b91a14f5d44f4eb21c0368c39ea492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39045 } }
I20260927 21:13:03.241348 9669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17b91a14f5d44f4eb21c0368c39ea492. Latest consensus state: current_term: 1 leader_uuid: "17b91a14f5d44f4eb21c0368c39ea492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b91a14f5d44f4eb21c0368c39ea492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39045 } } }
I20260927 21:13:03.241484 9669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [sys.catalog]: This master's current role is: LEADER
I20260927 21:13:03.241675 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.236286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.0.98.62:39045)
0927 21:13:03.236462 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:03.236469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:03.236500 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:13:03.236987 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:03.236995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:03.237069 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:03.237195 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:03.237481 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.238380 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.239447 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.239974 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.240073 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:03.240819 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:03.240823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:03.240823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:13:03.241089 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:03.241158 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:13:03.241175 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:03.241252 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:03.241389 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 21:13:03.241482 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260927 21:13:03.241348 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17b91a14f5d44f4eb21c0368c39ea492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b91a14f5d44f4eb21c0368c39ea492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39045 } } }
I20260927 21:13:03.241835 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492 [sys.catalog]: This master's current role is: LEADER
W20260927 21:13:03.242447 9683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:13:03.242512 9683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:13:03.242889 9677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:13:03.243067 9677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:13:03.244508 9677 catalog_manager.cc:1347] Generated new cluster ID: 8d579b12a0ff44efbedb9af4aaef86c8
I20260927 21:13:03.244563 9677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:13:03.251549 9677 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:13:03.252066 9677 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:13:03.261188 9677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17b91a14f5d44f4eb21c0368c39ea492: Generated new TSK 0
I20260927 21:13:03.261433 9677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:13:03.267988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39045
--builtin_ntp_servers=127.0.98.20:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 21:13:03.352775 9687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:03.352999 9687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:03.353031 9687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:03.354422 9687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:03.354485 9687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:03.354523 9687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:13:03.356030 9687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:03.356313 9687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:03.356629 9687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:03.362007 9693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:03.362020 9695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:03.362020 9692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.362248 9687 server_base.cc:1034] running on GCE node
I20260927 21:13:03.362558 9687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:03.362821 9687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:03.364017 9687 hybrid_clock.cc:648] HybridClock initialized: now 1790543583364000 us; error 32 us; skew 500 ppm
I20260927 21:13:03.365345 9687 webserver.cc:460] Webserver started at http://127.0.98.1:41295/ using document root <none> and password file <none>
I20260927 21:13:03.365597 9687 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:03.365649 9687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:03.365777 9687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:03.366891 9687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "25f48f0be66e4ae19a960056fd13b064"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "daf27ed935b4bb904a24f6d9c9d911f7"
server_key_iv: "3e164bf0f889cdb095e0cfe87e6cedb1"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.367249 9687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "25f48f0be66e4ae19a960056fd13b064"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "daf27ed935b4bb904a24f6d9c9d911f7"
server_key_iv: "3e164bf0f889cdb095e0cfe87e6cedb1"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.368633 9687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:13:03.369436 9701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.369575 9687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:13:03.369661 9687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "25f48f0be66e4ae19a960056fd13b064"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "daf27ed935b4bb904a24f6d9c9d911f7"
server_key_iv: "3e164bf0f889cdb095e0cfe87e6cedb1"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.369735 9687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:03.394037 9687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:03.394363 9687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:03.394502 9687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:03.394742 9687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:03.395305 9687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:03.395371 9687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.395408 9687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:03.395426 9687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.400988 9687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46139
I20260927 21:13:03.401060 9814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46139 every 8 connection(s)
I20260927 21:13:03.401401 9687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:13:03.403256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 9687
I20260927 21:13:03.403344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:13:03.403558 392 external_mini_cluster.cc:1428] Setting key f0d854f31f9e91ba600edcf3e3f33bdd
I20260927 21:13:03.404497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39045
--builtin_ntp_servers=127.0.98.20:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 21:13:03.407063 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.402241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41031 (local address 127.0.98.62:39045)
0927 21:13:03.402308 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:03.402309 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:03.402546 (+ 237us) server_negotiation.cc:407] Connection header received
0927 21:13:03.403039 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:03.403040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:03.403105 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:03.403175 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:03.404088 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.404584 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.405686 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.405887 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.405946 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:03.406798 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:03.406799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:03.406800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:13:03.406805 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:03.406821 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:13:03.406825 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:03.406872 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:03.406950 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 21:13:03.406980 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:13:03.407289 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.402161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39045 (local address 127.0.98.1:41031)
0927 21:13:03.402401 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:03.402447 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:03.402925 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:03.403297 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:03.403301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:03.403313 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:03.404004 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:03.404010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.404691 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.404692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:03.405600 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:03.405604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.405968 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.405969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:03.406098 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:03.406660 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:13:03.406660 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:03.406692 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:13:03.406717 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:03.406879 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:03.406883 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:03.406887 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:13:03.406957 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:13:03.407073 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
I20260927 21:13:03.407775 9815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39045
I20260927 21:13:03.407889 9815 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:03.408118 9815 heartbeater.cc:511] Master 127.0.98.62:39045 requested a full tablet report, sending...
I20260927 21:13:03.408722 9627 ts_manager.cc:194] Registered new tserver with Master: 25f48f0be66e4ae19a960056fd13b064 (127.0.98.1:46139)
I20260927 21:13:03.409711 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41031
W20260927 21:13:03.489674 9818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:03.489910 9818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:03.489933 9818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:03.491181 9818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:03.491225 9818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:03.491250 9818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:13:03.492648 9818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:03.492884 9818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:03.493160 9818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:03.498225 9824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.498359 9818 server_base.cc:1034] running on GCE node
W20260927 21:13:03.498230 9823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:03.498433 9826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.498708 9818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:03.498962 9818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:03.500128 9818 hybrid_clock.cc:648] HybridClock initialized: now 1790543583500110 us; error 30 us; skew 500 ppm
I20260927 21:13:03.501555 9818 webserver.cc:460] Webserver started at http://127.0.98.2:42689/ using document root <none> and password file <none>
I20260927 21:13:03.501791 9818 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:03.501842 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:03.501983 9818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:03.503098 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "347325bed0524519baaf9e4a697c4415"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "3d746653bea30ee729d178c2c4de4f30"
server_key_iv: "220cd28bbbb2558b1e8dfb1342b62d87"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.503448 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "347325bed0524519baaf9e4a697c4415"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "3d746653bea30ee729d178c2c4de4f30"
server_key_iv: "220cd28bbbb2558b1e8dfb1342b62d87"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.504815 9818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:13:03.505601 9832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.505757 9818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:13:03.505843 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "347325bed0524519baaf9e4a697c4415"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "3d746653bea30ee729d178c2c4de4f30"
server_key_iv: "220cd28bbbb2558b1e8dfb1342b62d87"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.505910 9818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:03.514837 9818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:03.515203 9818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:03.515347 9818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:03.515568 9818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:03.515903 9818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:03.515936 9818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.515966 9818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:03.515990 9818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.521719 9818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34707
I20260927 21:13:03.521765 9945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34707 every 8 connection(s)
I20260927 21:13:03.522094 9818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:13:03.527341 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.522782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50393 (local address 127.0.98.62:39045)
0927 21:13:03.522847 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:03.522849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:03.523096 (+ 247us) server_negotiation.cc:407] Connection header received
0927 21:13:03.523568 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:03.523569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:03.523580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:03.523638 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:03.524289 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.524796 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.525948 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.526148 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.526223 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:03.527093 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:03.527095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:03.527095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:13:03.527101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:03.527120 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:13:03.527124 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:03.527169 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:03.527236 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 21:13:03.527262 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:13:03.527537 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.522793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39045 (local address 127.0.98.2:50393)
0927 21:13:03.522965 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:03.523000 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:03.523481 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:03.523698 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:03.523701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:03.523710 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:03.524200 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:03.524204 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.524896 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.524897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:03.525845 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:03.525848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.526250 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.526251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:03.526400 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:03.526953 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:13:03.526954 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:03.526984 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:13:03.527008 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:03.527176 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:03.527178 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:03.527181 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:13:03.527230 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:13:03.527331 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260927 21:13:03.527980 9946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39045
I20260927 21:13:03.528079 9946 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:03.528254 9946 heartbeater.cc:511] Master 127.0.98.62:39045 requested a full tablet report, sending...
I20260927 21:13:03.528575 9627 ts_manager.cc:194] Registered new tserver with Master: 347325bed0524519baaf9e4a697c4415 (127.0.98.2:34707)
I20260927 21:13:03.528942 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50393
I20260927 21:13:03.529371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 9818
I20260927 21:13:03.529467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:13:03.529666 392 external_mini_cluster.cc:1428] Setting key 175e4c79948924cd03fb52e8eef4651a
I20260927 21:13:03.530421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39045
--builtin_ntp_servers=127.0.98.20:42473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 21:13:03.613875 9949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:03.614108 9949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:03.614132 9949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:03.615453 9949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:03.615509 9949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:03.615535 9949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:13:03.617214 9949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:03.617513 9949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:03.617795 9949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:03.623044 9955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:03.623034 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.623124 9949 server_base.cc:1034] running on GCE node
W20260927 21:13:03.623034 9954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.623474 9949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:03.623728 9949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:03.624897 9949 hybrid_clock.cc:648] HybridClock initialized: now 1790543583624881 us; error 33 us; skew 500 ppm
I20260927 21:13:03.626255 9949 webserver.cc:460] Webserver started at http://127.0.98.3:36713/ using document root <none> and password file <none>
I20260927 21:13:03.626503 9949 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:03.626556 9949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:03.626689 9949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:03.627792 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "ff9c602f7fda482ebed958cbf9485228"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "c6fe8ab6b9cb3961a77cbf28a73cb6da"
server_key_iv: "262cf4d6fee75c8c027c6344dbc63a8a"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.628125 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ff9c602f7fda482ebed958cbf9485228"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "c6fe8ab6b9cb3961a77cbf28a73cb6da"
server_key_iv: "262cf4d6fee75c8c027c6344dbc63a8a"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.629458 9949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:13:03.630255 9963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.630429 9949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:13:03.630512 9949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "ff9c602f7fda482ebed958cbf9485228"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "c6fe8ab6b9cb3961a77cbf28a73cb6da"
server_key_iv: "262cf4d6fee75c8c027c6344dbc63a8a"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.630576 9949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:03.645701 9949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:03.646041 9949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:03.646174 9949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:03.646391 9949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:03.646701 9949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:03.646736 9949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.646767 9949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:03.646791 9949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.652426 9949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43907
I20260927 21:13:03.652511 10076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43907 every 8 connection(s)
I20260927 21:13:03.652812 9949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:13:03.655243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 9949
I20260927 21:13:03.655339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:13:03.655524 392 external_mini_cluster.cc:1428] Setting key ecd4a09c93e1134b8d5695028d169cf0
I20260927 21:13:03.658650 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.653766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60461 (local address 127.0.98.62:39045)
0927 21:13:03.653854 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:03.653856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:03.653961 (+ 105us) server_negotiation.cc:407] Connection header received
0927 21:13:03.654494 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:03.654495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:03.654508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:03.654556 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:03.655293 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.655969 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.657167 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.657394 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.657453 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:03.658301 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:03.658302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:03.658302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:13:03.658307 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:03.658324 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:13:03.658328 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:03.658389 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:03.658556 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 21:13:03.658582 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:13:03.658880 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.653644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39045 (local address 127.0.98.3:60461)
0927 21:13:03.653824 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:03.653870 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:03.654382 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:03.654641 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:03.654646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:03.654657 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:03.655201 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:03.655206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.656076 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.656077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:03.657063 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:03.657066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:03.657477 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:03.657478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:03.657600 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:03.658147 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:13:03.658147 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:03.658179 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:13:03.658207 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:03.658431 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:03.658438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:03.658441 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:13:03.658520 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 21:13:03.658638 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260927 21:13:03.659493 10077 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39045
I20260927 21:13:03.659612 10077 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:03.659802 10077 heartbeater.cc:511] Master 127.0.98.62:39045 requested a full tablet report, sending...
I20260927 21:13:03.660254 9627 ts_manager.cc:194] Registered new tserver with Master: ff9c602f7fda482ebed958cbf9485228 (127.0.98.3:43907)
I20260927 21:13:03.660635 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60461
I20260927 21:13:03.667605 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:13:03.669508 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:03.668974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.0.98.62:39045)
0927 21:13:03.669032 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:03.669034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:03.669141 (+ 107us) server_negotiation.cc:407] Connection header received
0927 21:13:03.669206 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:03.669207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:03.669217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:03.669292 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:03.669447 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36358: BlockingRecv error: recv got EOF from 127.0.0.1:36358 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:13:03.669657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 9687
I20260927 21:13:03.673889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 9818
I20260927 21:13:03.678030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 9949
I20260927 21:13:03.682077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 9596
2026-09-27T21:13:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (587 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T21:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:13:03Z Disabled control of system clock
I20260927 21:13:03.708221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42701
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:13:03.790412 10088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:03.790637 10088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:03.790659 10088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:03.791914 10088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:13:03.791966 10088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:03.791977 10088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:03.791989 10088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:13:03.792001 10088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:13:03.793488 10088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:03.793745 10088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:03.794139 10088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:03.799218 10096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:03.799288 10093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:03.799218 10094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.799366 10088 server_base.cc:1034] running on GCE node
I20260927 21:13:03.799784 10088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:03.800066 10088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:03.801245 10088 hybrid_clock.cc:648] HybridClock initialized: now 1790543583801226 us; error 35 us; skew 500 ppm
I20260927 21:13:03.802666 10088 webserver.cc:460] Webserver started at http://127.0.98.62:38281/ using document root <none> and password file <none>
I20260927 21:13:03.802894 10088 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:03.802947 10088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:03.803073 10088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:03.804145 10088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "6b2db70aa9934c4fbb0d8f1576934744"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "53b6925f2f9eae34f641a7acd0eb7105"
server_key_iv: "389a3eb7e0d4ad3bc32c40d07801adf1"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.804493 10088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b2db70aa9934c4fbb0d8f1576934744"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "53b6925f2f9eae34f641a7acd0eb7105"
server_key_iv: "389a3eb7e0d4ad3bc32c40d07801adf1"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.805866 10088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:13:03.806625 10102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.806829 10088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:13:03.806914 10088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "6b2db70aa9934c4fbb0d8f1576934744"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "53b6925f2f9eae34f641a7acd0eb7105"
server_key_iv: "389a3eb7e0d4ad3bc32c40d07801adf1"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.806980 10088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:03.810433 10088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:03.810698 10088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:03.810823 10088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:03.814607 10088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42701
I20260927 21:13:03.814653 10154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42701 every 8 connection(s)
I20260927 21:13:03.814972 10088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:13:03.815619 10155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:03.818419 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744: Bootstrap starting.
I20260927 21:13:03.818991 10155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:03.819255 10155 log.cc:824] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:03.819916 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744: No bootstrap required, opened a new log
I20260927 21:13:03.821575 10155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2db70aa9934c4fbb0d8f1576934744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } }
I20260927 21:13:03.821731 10155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:03.821753 10155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2db70aa9934c4fbb0d8f1576934744, State: Initialized, Role: FOLLOWER
I20260927 21:13:03.821898 10155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2db70aa9934c4fbb0d8f1576934744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } }
I20260927 21:13:03.821970 10155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:13:03.821992 10155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:13:03.822024 10155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:03.822592 10155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2db70aa9934c4fbb0d8f1576934744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } }
I20260927 21:13:03.822679 10155 leader_election.cc:302] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b2db70aa9934c4fbb0d8f1576934744; no voters:
I20260927 21:13:03.822841 10155 leader_election.cc:288] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:13:03.822824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 10088
I20260927 21:13:03.822911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:13:03.822933 10158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:13:03.823092 10158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [term 1 LEADER]: Becoming Leader. State: Replica: 6b2db70aa9934c4fbb0d8f1576934744, State: Running, Role: LEADER
I20260927 21:13:03.823109 392 external_mini_cluster.cc:1428] Setting key 799cb87505b4841edc6b8d86fac15b2f
I20260927 21:13:03.823108 10155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:13:03.823280 10158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2db70aa9934c4fbb0d8f1576934744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } }
I20260927 21:13:03.823879 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b2db70aa9934c4fbb0d8f1576934744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2db70aa9934c4fbb0d8f1576934744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } } }
I20260927 21:13:03.823974 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [sys.catalog]: This master's current role is: LEADER
I20260927 21:13:03.824172 10160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b2db70aa9934c4fbb0d8f1576934744. Latest consensus state: current_term: 1 leader_uuid: "6b2db70aa9934c4fbb0d8f1576934744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2db70aa9934c4fbb0d8f1576934744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } } }
I20260927 21:13:03.824198 10169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:13:03.824262 10160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744 [sys.catalog]: This master's current role is: LEADER
I20260927 21:13:03.824812 10169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:13:03.825963 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.823596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.0.98.62:42701)
0927 21:13:03.823778 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:03.823789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:03.823825 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:13:03.823949 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:03.823956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:03.824007 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:03.825032 (+ 1025us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:03.825493 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:03.825496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:03.825496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:13:03.825499 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:03.825541 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:13:03.825553 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:03.825646 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:03.825716 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 21:13:03.825776 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260927 21:13:03.826632 10169 catalog_manager.cc:1347] Generated new cluster ID: 4eb54fbb379742f596aead287e33af34
I20260927 21:13:03.826794 10169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:13:03.847029 10169 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:13:03.847245 10169 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:13:03.858284 10169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b2db70aa9934c4fbb0d8f1576934744: Generated new TSK 0
I20260927 21:13:03.858497 10169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:13:03.870174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--builtin_ntp_servers=127.0.98.20:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:13:03.953812 10179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:03.954034 10179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:03.954066 10179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:03.955289 10179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:03.955382 10179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:03.955438 10179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:13:03.956877 10179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:03.957091 10179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:03.957372 10179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:03.962357 10184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:03.962425 10185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.962499 10179 server_base.cc:1034] running on GCE node
W20260927 21:13:03.962435 10187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:03.962795 10179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:03.963023 10179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:03.964193 10179 hybrid_clock.cc:648] HybridClock initialized: now 1790543583964179 us; error 29 us; skew 500 ppm
I20260927 21:13:03.965518 10179 webserver.cc:460] Webserver started at http://127.0.98.1:33519/ using document root <none> and password file <none>
I20260927 21:13:03.965752 10179 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:03.965796 10179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:03.965891 10179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:03.966938 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "5e01464407f14ed682caf958a74a8565"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "6214a2b32b8480a32fe82b8cc1f760b7"
server_key_iv: "3d85382e312f3f779d784170e0eeedc9"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.967248 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5e01464407f14ed682caf958a74a8565"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "6214a2b32b8480a32fe82b8cc1f760b7"
server_key_iv: "3d85382e312f3f779d784170e0eeedc9"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.968410 10179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:13:03.969252 10193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.969442 10179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:13:03.969547 10179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "5e01464407f14ed682caf958a74a8565"
format_stamp: "Formatted at 2026-09-27 21:13:03 on dist-test-slave-r9x1"
server_key: "6214a2b32b8480a32fe82b8cc1f760b7"
server_key_iv: "3d85382e312f3f779d784170e0eeedc9"
server_key_version: "encryptionkey@0"
I20260927 21:13:03.969627 10179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:03.973112 10179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:03.973453 10179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:03.973562 10179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:03.973752 10179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:03.974047 10179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:03.974077 10179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.974107 10179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:03.974131 10179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:03.979830 10179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32819
I20260927 21:13:03.979879 10306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32819 every 8 connection(s)
I20260927 21:13:03.980229 10179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:13:03.983328 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.981059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59047 (local address 127.0.98.62:42701)
0927 21:13:03.981109 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:03.981111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:03.981247 (+ 136us) server_negotiation.cc:407] Connection header received
0927 21:13:03.981768 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:03.981773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:03.981780 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:03.981843 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:03.983047 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:03.983048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:03.983048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:13:03.983052 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:03.983067 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:13:03.983076 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:03.983136 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:03.983247 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:13:03.983267 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:13:03.983526 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:03.980936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42701 (local address 127.0.98.1:59047)
0927 21:13:03.981123 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:03.981166 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:03.981679 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:03.981899 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:03.981902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:03.981911 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:03.982846 (+ 935us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:13:03.982846 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:03.982891 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:13:03.982926 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:03.983165 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:03.983171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:03.983174 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:13:03.983259 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:13:03.983318 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260927 21:13:03.983979 10307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42701
I20260927 21:13:03.984069 10307 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:03.984184 10307 heartbeater.cc:511] Master 127.0.98.62:42701 requested a full tablet report, sending...
I20260927 21:13:03.984650 10119 ts_manager.cc:194] Registered new tserver with Master: 5e01464407f14ed682caf958a74a8565 (127.0.98.1:32819)
I20260927 21:13:03.986032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 10179
I20260927 21:13:03.986115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:13:03.986322 392 external_mini_cluster.cc:1428] Setting key 483e889901aeaa8905c201a6ebdd4a9d
I20260927 21:13:03.987295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--builtin_ntp_servers=127.0.98.20:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:13:04.067811 10310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:04.068075 10310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:04.068099 10310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:04.069368 10310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:04.069415 10310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:04.069440 10310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:13:04.070832 10310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:04.071055 10310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:04.071316 10310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:04.076368 10316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:04.076408 10315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:04.076380 10318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:04.076419 10310 server_base.cc:1034] running on GCE node
I20260927 21:13:04.076825 10310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:04.077093 10310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:04.078235 10310 hybrid_clock.cc:648] HybridClock initialized: now 1790543584078207 us; error 47 us; skew 500 ppm
I20260927 21:13:04.079582 10310 webserver.cc:460] Webserver started at http://127.0.98.2:43891/ using document root <none> and password file <none>
I20260927 21:13:04.079808 10310 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:04.079862 10310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:04.080022 10310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:04.081133 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "43c4263100994c75a9725b59bcc7a13e"
format_stamp: "Formatted at 2026-09-27 21:13:04 on dist-test-slave-r9x1"
server_key: "4fda03fb4e837ac48e9818864717cbca"
server_key_iv: "10f1c9c72ffe86f8c59a34eae77e9b75"
server_key_version: "encryptionkey@0"
I20260927 21:13:04.081533 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43c4263100994c75a9725b59bcc7a13e"
format_stamp: "Formatted at 2026-09-27 21:13:04 on dist-test-slave-r9x1"
server_key: "4fda03fb4e837ac48e9818864717cbca"
server_key_iv: "10f1c9c72ffe86f8c59a34eae77e9b75"
server_key_version: "encryptionkey@0"
I20260927 21:13:04.082906 10310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:13:04.083666 10324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:04.083827 10310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:13:04.083915 10310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "43c4263100994c75a9725b59bcc7a13e"
format_stamp: "Formatted at 2026-09-27 21:13:04 on dist-test-slave-r9x1"
server_key: "4fda03fb4e837ac48e9818864717cbca"
server_key_iv: "10f1c9c72ffe86f8c59a34eae77e9b75"
server_key_version: "encryptionkey@0"
I20260927 21:13:04.083994 10310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:04.087563 10310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:04.087845 10310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:04.087971 10310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:04.088173 10310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:04.088478 10310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:04.088510 10310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:04.088539 10310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:04.088562 10310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:04.094417 10310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36017
I20260927 21:13:04.094465 10437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36017 every 8 connection(s)
I20260927 21:13:04.094784 10310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:13:04.097702 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:04.095464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44019 (local address 127.0.98.62:42701)
0927 21:13:04.095518 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:04.095521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:04.095767 (+ 246us) server_negotiation.cc:407] Connection header received
0927 21:13:04.096190 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:04.096192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:04.096197 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:04.096282 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:04.097494 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:04.097495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:04.097496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:13:04.097500 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:04.097516 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:13:04.097521 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:04.097580 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:04.097631 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 21:13:04.097642 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:13:04.097918 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:04.095428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42701 (local address 127.0.98.2:44019)
0927 21:13:04.095609 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:04.095650 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:04.096110 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:04.096370 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:04.096373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:04.096382 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:04.097331 (+ 949us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:13:04.097332 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:04.097369 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:13:04.097402 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:04.097582 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:04.097584 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:04.097587 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:13:04.097640 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:13:04.097719 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260927 21:13:04.098466 10438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42701
I20260927 21:13:04.098562 10438 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:04.098717 10438 heartbeater.cc:511] Master 127.0.98.62:42701 requested a full tablet report, sending...
I20260927 21:13:04.099071 10119 ts_manager.cc:194] Registered new tserver with Master: 43c4263100994c75a9725b59bcc7a13e (127.0.98.2:36017)
I20260927 21:13:04.102046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 10310
I20260927 21:13:04.102134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:13:04.102280 392 external_mini_cluster.cc:1428] Setting key 65f029d164a950eea4b232ac6d3de1e0
I20260927 21:13:04.103176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--builtin_ntp_servers=127.0.98.20:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:13:04.184000 10441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:04.184245 10441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:04.184283 10441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:04.185632 10441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:04.185688 10441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:04.185724 10441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:13:04.187279 10441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:04.187551 10441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:04.187871 10441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:04.193063 10447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:04.193228 10449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:04.193324 10441 server_base.cc:1034] running on GCE node
W20260927 21:13:04.193064 10446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:04.193616 10441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:04.193871 10441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:04.195021 10441 hybrid_clock.cc:648] HybridClock initialized: now 1790543584195007 us; error 33 us; skew 500 ppm
I20260927 21:13:04.196309 10441 webserver.cc:460] Webserver started at http://127.0.98.3:34961/ using document root <none> and password file <none>
I20260927 21:13:04.196537 10441 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:04.196583 10441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:04.196707 10441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:04.197832 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "a72c48baae4c49a085c390ab39246371"
format_stamp: "Formatted at 2026-09-27 21:13:04 on dist-test-slave-r9x1"
server_key: "c2d4435ca030e20cc43cc789a587c6e4"
server_key_iv: "14518b76a4204cccd36e7ff0080c66b4"
server_key_version: "encryptionkey@0"
I20260927 21:13:04.198171 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a72c48baae4c49a085c390ab39246371"
format_stamp: "Formatted at 2026-09-27 21:13:04 on dist-test-slave-r9x1"
server_key: "c2d4435ca030e20cc43cc789a587c6e4"
server_key_iv: "14518b76a4204cccd36e7ff0080c66b4"
server_key_version: "encryptionkey@0"
I20260927 21:13:04.199430 10441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:13:04.200160 10455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:04.200306 10441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:13:04.200385 10441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "a72c48baae4c49a085c390ab39246371"
format_stamp: "Formatted at 2026-09-27 21:13:04 on dist-test-slave-r9x1"
server_key: "c2d4435ca030e20cc43cc789a587c6e4"
server_key_iv: "14518b76a4204cccd36e7ff0080c66b4"
server_key_version: "encryptionkey@0"
I20260927 21:13:04.200459 10441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:04.204084 10441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:04.204368 10441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:04.204492 10441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:04.204685 10441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:04.204986 10441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:04.205019 10441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:04.205040 10441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:04.205063 10441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:04.211310 10441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33543
I20260927 21:13:04.211369 10568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33543 every 8 connection(s)
I20260927 21:13:04.211777 10441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:13:04.215253 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:04.212716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59767 (local address 127.0.98.62:42701)
0927 21:13:04.212764 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:04.212766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:04.213028 (+ 262us) server_negotiation.cc:407] Connection header received
0927 21:13:04.213606 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:04.213607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:04.213612 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:04.213666 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:04.214984 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:04.214986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:04.214986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:13:04.214993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:04.215007 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:13:04.215012 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:04.215062 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:04.215181 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 21:13:04.215194 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:13:04.215487 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:04.212726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42701 (local address 127.0.98.3:59767)
0927 21:13:04.212908 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:04.212945 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:04.213510 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:04.213724 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:04.213727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:04.213736 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:04.214734 (+ 998us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:13:04.214734 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:04.214776 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:13:04.214809 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:04.215090 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:04.215097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:04.215100 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:13:04.215186 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:13:04.215247 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260927 21:13:04.215966 10569 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42701
I20260927 21:13:04.216061 10569 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:04.216212 10569 heartbeater.cc:511] Master 127.0.98.62:42701 requested a full tablet report, sending...
I20260927 21:13:04.216552 10119 ts_manager.cc:194] Registered new tserver with Master: a72c48baae4c49a085c390ab39246371 (127.0.98.3:33543)
I20260927 21:13:04.218037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 10441
I20260927 21:13:04.218122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:13:04.218281 392 external_mini_cluster.cc:1428] Setting key e8fe69768a1ac826ee16eda38fadecce
I20260927 21:13:04.219101 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:13:04.220788 10163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:13:04.220364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.0.98.62:42701)
0927 21:13:04.220413 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:04.220416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:04.220514 (+ 98us) server_negotiation.cc:407] Connection header received
0927 21:13:04.220560 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:04.220561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:04.220565 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:04.220609 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:04.220721 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33592: BlockingRecv error: recv got EOF from 127.0.0.1:33592 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:13:04.220834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 10179
I20260927 21:13:04.225956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 10310
I20260927 21:13:04.230288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 10441
I20260927 21:13:04.234247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 10088
2026-09-27T21:13:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (551 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:13:04 dist-test-slave-r9x1 krb5kdc[10579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:13:04 dist-test-slave-r9x1 krb5kdc[10579](info): set up 2 sockets
Sep 27 21:13:04 dist-test-slave-r9x1 krb5kdc[10579](info): commencing operation
krb5kdc: starting...
W20260927 21:13:05.194085 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:13:05 dist-test-slave-r9x1 krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:13:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:13:05.312422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:13:05.396345 10592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:05.396591 10592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:05.396615 10592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:05.397962 10592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:13:05.398023 10592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:05.398036 10592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:05.398051 10592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:13:05.398061 10592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:13:05.399727 10592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:05.400002 10592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:05.400228 10592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:05.405258 10598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:05.405260 10600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:05.405318 10592 server_base.cc:1034] running on GCE node
W20260927 21:13:05.405260 10597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:05.405748 10592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:05.406019 10592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:05.407169 10592 hybrid_clock.cc:648] HybridClock initialized: now 1790543585407149 us; error 40 us; skew 500 ppm
Sep 27 21:13:05 dist-test-slave-r9x1 krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:13:05.409648 10592 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:13:05.410097 10592 webserver.cc:460] Webserver started at http://127.0.98.62:37001/ using document root <none> and password file <none>
I20260927 21:13:05.410288 10592 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:05.410331 10592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:05.410424 10592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:05.411500 10592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "9084cfb076494f439fc2164900e69ab7"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "92e27544333d60cee5eca72a4ee9851b"
server_key_iv: "e28ebd935d6ed03696b86b4c47533208"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.411893 10592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "9084cfb076494f439fc2164900e69ab7"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "92e27544333d60cee5eca72a4ee9851b"
server_key_iv: "e28ebd935d6ed03696b86b4c47533208"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.413309 10592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:13:05.414188 10607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.414371 10592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:13:05.414463 10592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "9084cfb076494f439fc2164900e69ab7"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "92e27544333d60cee5eca72a4ee9851b"
server_key_iv: "e28ebd935d6ed03696b86b4c47533208"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.414523 10592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:05.431166 10592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:05.433426 10592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:05.433574 10592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:05.437354 10592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34765
I20260927 21:13:05.437412 10659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34765 every 8 connection(s)
I20260927 21:13:05.437764 10592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:13:05.438748 10660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:05.441059 10660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7: Bootstrap starting.
I20260927 21:13:05.441679 10660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:05.441941 10660 log.cc:824] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:05.442736 10660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7: No bootstrap required, opened a new log
I20260927 21:13:05.444448 10660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9084cfb076494f439fc2164900e69ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } }
I20260927 21:13:05.444620 10660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:05.444653 10660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9084cfb076494f439fc2164900e69ab7, State: Initialized, Role: FOLLOWER
I20260927 21:13:05.444803 10660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9084cfb076494f439fc2164900e69ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } }
I20260927 21:13:05.444887 10660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:13:05.444921 10660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:13:05.444970 10660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:05.445940 10660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9084cfb076494f439fc2164900e69ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } }
I20260927 21:13:05.446049 10660 leader_election.cc:302] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9084cfb076494f439fc2164900e69ab7; no voters:
I20260927 21:13:05.446220 10660 leader_election.cc:288] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:13:05.446303 10663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:13:05.446466 10663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [term 1 LEADER]: Becoming Leader. State: Replica: 9084cfb076494f439fc2164900e69ab7, State: Running, Role: LEADER
I20260927 21:13:05.446548 10660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:13:05.446565 10663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9084cfb076494f439fc2164900e69ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } }
I20260927 21:13:05.446920 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9084cfb076494f439fc2164900e69ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9084cfb076494f439fc2164900e69ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } }
I20260927 21:13:05.447010 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:13:05.446944 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9084cfb076494f439fc2164900e69ab7. Latest consensus state: current_term: 1 leader_uuid: "9084cfb076494f439fc2164900e69ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9084cfb076494f439fc2164900e69ab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } }
I20260927 21:13:05.447190 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:13:05.447346 10669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:13:05.447677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 10592
I20260927 21:13:05.447773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:13:05.447903 10669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:13:05.447966 392 external_mini_cluster.cc:1428] Setting key b8c85f6e19174ae4cfc68d0064c3af31
I20260927 21:13:05.449702 10669 catalog_manager.cc:1347] Generated new cluster ID: 3d0a7a9c33c24a33be8173f9aa3b884f
I20260927 21:13:05.449762 10669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:13:05.454406 10669 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:13:05.454907 10669 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 21:13:05 dist-test-slave-r9x1 krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:13:05.459964 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.448405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.0.98.62:34765)
0927 21:13:05.448662 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:05.448672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:05.448706 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:13:05.448826 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:05.448833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:05.448898 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:05.449040 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:05.449876 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.451263 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.452369 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.453070 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.453179 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.456195 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:05.456200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:05.456201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:05.456216 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:05.458223 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.458635 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.458639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.458639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.458763 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.458987 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.458987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.458988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.459311 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:05.459474 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:05.459549 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 21:13:05.459709 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":68,"threads_started":1}
I20260927 21:13:05.460430 10669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7: Generated new TSK 0
I20260927 21:13:05.460593 10669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:13:05.496483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34765
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:13:05.583340 10686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:05.583580 10686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:05.583606 10686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:05.584894 10686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:05.584949 10686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:05.584976 10686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:13:05.586617 10686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:05.586843 10686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:05.587044 10686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:13:05.592137 10686 server_base.cc:1034] running on GCE node
W20260927 21:13:05.592154 10691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:05.592226 10692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:05.592468 10694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:05.592756 10686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:05.593029 10686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:05.594184 10686 hybrid_clock.cc:648] HybridClock initialized: now 1790543585594168 us; error 32 us; skew 500 ppm
Sep 27 21:13:05 dist-test-slave-r9x1 krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:13:05.597024 10686 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:13:05.597555 10686 webserver.cc:460] Webserver started at http://127.0.98.1:40227/ using document root <none> and password file <none>
I20260927 21:13:05.597774 10686 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:05.597824 10686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:05.597949 10686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:05.599056 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "c9401be9f2854fadb018e1cd275b7f2b"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "82281906dfcd5e3ece2f53ed46b84bcf"
server_key_iv: "7ab78661f59bece1ee98de6fafd485a7"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.599409 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c9401be9f2854fadb018e1cd275b7f2b"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "82281906dfcd5e3ece2f53ed46b84bcf"
server_key_iv: "7ab78661f59bece1ee98de6fafd485a7"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.600759 10686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:13:05.601502 10701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.601747 10686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:13:05.601835 10686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "c9401be9f2854fadb018e1cd275b7f2b"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "82281906dfcd5e3ece2f53ed46b84bcf"
server_key_iv: "7ab78661f59bece1ee98de6fafd485a7"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.601895 10686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:05.616578 10686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:05.618880 10686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:05.619014 10686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:05.619215 10686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:05.619527 10686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:05.619565 10686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.619586 10686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:05.619597 10686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.625240 10686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38451
I20260927 21:13:05.625407 10814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38451 every 8 connection(s)
I20260927 21:13:05.625646 10686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:13:05 dist-test-slave-r9x1 krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:13:05.632337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 10686
I20260927 21:13:05.632444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:13:05.632660 392 external_mini_cluster.cc:1428] Setting key a802332cf5e77414e40579c76c9261e5
I20260927 21:13:05.635334 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.626349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58463 (local address 127.0.98.62:34765)
0927 21:13:05.626401 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:05.626403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:05.626602 (+ 199us) server_negotiation.cc:407] Connection header received
0927 21:13:05.627184 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:05.627185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:05.627199 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:05.627259 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:05.628233 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.628745 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.629790 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.630018 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.630074 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.632292 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:05.632293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:05.632293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:05.632303 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:05.633794 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.634225 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.634226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.634227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.634288 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.634539 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.634540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.634541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.634692 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:05.634735 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:05.635173 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 21:13:05.635265 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:13:05.635529 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.626321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34765 (local address 127.0.98.1:58463)
0927 21:13:05.626475 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:05.626515 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:05.627071 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:05.627331 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:05.627336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:05.627928 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:05.628121 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.628126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.628836 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.628837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.629693 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.629696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.630095 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.630096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.630218 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.630701 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:05.630736 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:05.632184 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:05.633878 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:05.633887 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:05.633890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:05.634152 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:05.634345 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:05.634346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:05.634346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:05.634463 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:05.634780 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:05.634787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:05.634973 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 21:13:05.635102 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 21:13:05.635290 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260927 21:13:05.636207 10815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34765
I20260927 21:13:05.636329 10815 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:05.636541 10815 heartbeater.cc:511] Master 127.0.98.62:34765 requested a full tablet report, sending...
I20260927 21:13:05.637135 10624 ts_manager.cc:194] Registered new tserver with Master: c9401be9f2854fadb018e1cd275b7f2b (127.0.98.1:38451)
I20260927 21:13:05.637813 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58463
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:13:05.661113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34765
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:13:05.742029 10820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:05.742242 10820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:05.742266 10820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:05.743471 10820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:05.743530 10820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:05.743557 10820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:13:05.745085 10820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:05.745347 10820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:05.745584 10820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:13:05.750780 10820 server_base.cc:1034] running on GCE node
W20260927 21:13:05.750833 10828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:05.750765 10825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:05.750744 10826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:05.751232 10820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:05.751488 10820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:05.752640 10820 hybrid_clock.cc:648] HybridClock initialized: now 1790543585752624 us; error 32 us; skew 500 ppm
Sep 27 21:13:05 dist-test-slave-r9x1 krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:13:05.755184 10820 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:13:05.755658 10820 webserver.cc:460] Webserver started at http://127.0.98.2:38255/ using document root <none> and password file <none>
I20260927 21:13:05.755874 10820 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:05.755921 10820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:05.756049 10820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:05.757076 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "289579b24bc0ab919bca99e8ac18b863"
server_key_iv: "b67f00883aeec21b8255d43df708aa19"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.757423 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "289579b24bc0ab919bca99e8ac18b863"
server_key_iv: "b67f00883aeec21b8255d43df708aa19"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.758796 10820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:13:05.759691 10835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.759881 10820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:13:05.759961 10820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "289579b24bc0ab919bca99e8ac18b863"
server_key_iv: "b67f00883aeec21b8255d43df708aa19"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.760035 10820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:05.775956 10820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:05.778044 10820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:05.778191 10820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:05.778417 10820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:05.778728 10820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:05.778760 10820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.778798 10820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:05.778820 10820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.784363 10820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34045
I20260927 21:13:05.784412 10948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34045 every 8 connection(s)
I20260927 21:13:05.784737 10820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:13:05.786052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 10820
I20260927 21:13:05.786152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:13:05.786360 392 external_mini_cluster.cc:1428] Setting key 02bf539861ea81bbb1e0b3c286329249
Sep 27 21:13:05 dist-test-slave-r9x1 krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:13:05.795296 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.785511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52177 (local address 127.0.98.62:34765)
0927 21:13:05.785584 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:05.785586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:05.785763 (+ 177us) server_negotiation.cc:407] Connection header received
0927 21:13:05.786295 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:05.786297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:05.786311 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:05.786367 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:05.787410 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.788083 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.789518 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.789737 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.789814 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.792547 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:05.792548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:05.792549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:05.792556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:05.793827 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.794313 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.794314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.794314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.794373 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.794669 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.794670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.794670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.794795 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:05.794849 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:05.795149 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 21:13:05.795227 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:13:05.795653 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.785484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34765 (local address 127.0.98.2:52177)
0927 21:13:05.785648 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:05.785686 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:05.786171 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:05.786442 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:05.786447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:05.787117 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:05.787321 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.787326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.788160 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.788161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.789431 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.789434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.789910 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.789911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.790070 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.790690 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:05.790701 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:05.792405 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:05.793915 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:05.793923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:05.793926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:05.794222 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:05.794453 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:05.794453 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:05.794454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:05.794604 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:05.794881 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:05.794887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:05.795031 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 21:13:05.795131 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:13:05.795278 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260927 21:13:05.796170 10949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34765
I20260927 21:13:05.796264 10949 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:05.796470 10949 heartbeater.cc:511] Master 127.0.98.62:34765 requested a full tablet report, sending...
I20260927 21:13:05.796856 10624 ts_manager.cc:194] Registered new tserver with Master: 165af050b4ab4ae9be5cfc0ac9b4e9fc (127.0.98.2:34045)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 21:13:05.797253 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:13:05.812386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34765
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:13:05.893180 10954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:05.893426 10954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:05.893450 10954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:05.894589 10954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:05.894629 10954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:05.894652 10954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:13:05.896400 10954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:05.896663 10954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:05.896898 10954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:05.901890 10959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:05.901892 10962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:05.901886 10960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:05.902251 10954 server_base.cc:1034] running on GCE node
I20260927 21:13:05.902477 10954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:05.902720 10954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:05.903877 10954 hybrid_clock.cc:648] HybridClock initialized: now 1790543585903859 us; error 31 us; skew 500 ppm
Sep 27 21:13:05 dist-test-slave-r9x1 krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:13:05.906380 10954 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:13:05.906865 10954 webserver.cc:460] Webserver started at http://127.0.98.3:42813/ using document root <none> and password file <none>
I20260927 21:13:05.907061 10954 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:05.907119 10954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:05.907245 10954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:13:05.908252 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "b11f1a397cd649f4b663ce9728e80170"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "e10289b5fa8568c9d52096e97039e1e3"
server_key_iv: "62d40ce874f6cce36c57f81f4f93613c"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.908581 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b11f1a397cd649f4b663ce9728e80170"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "e10289b5fa8568c9d52096e97039e1e3"
server_key_iv: "62d40ce874f6cce36c57f81f4f93613c"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.909888 10954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:13:05.910679 10969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.910977 10954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:13:05.911073 10954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "b11f1a397cd649f4b663ce9728e80170"
format_stamp: "Formatted at 2026-09-27 21:13:05 on dist-test-slave-r9x1"
server_key: "e10289b5fa8568c9d52096e97039e1e3"
server_key_iv: "62d40ce874f6cce36c57f81f4f93613c"
server_key_version: "encryptionkey@0"
I20260927 21:13:05.911142 10954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:05.920848 10954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:05.922924 10954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:05.923070 10954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:05.923292 10954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:05.923597 10954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:05.923632 10954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.923662 10954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:05.923679 10954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:05.929340 10954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41283
I20260927 21:13:05.929392 11082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41283 every 8 connection(s)
I20260927 21:13:05.929705 10954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:13:05 dist-test-slave-r9x1 krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:13:05.937191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 10954
I20260927 21:13:05.937311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:13:05.937479 392 external_mini_cluster.cc:1428] Setting key cb28a39fd0af42e3ff0abcc35a13cbc9
I20260927 21:13:05.938932 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.930386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60689 (local address 127.0.98.62:34765)
0927 21:13:05.930464 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:05.930466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:05.930631 (+ 165us) server_negotiation.cc:407] Connection header received
0927 21:13:05.931113 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:05.931114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:05.931125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:05.931178 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:05.932231 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.932755 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.933823 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.934021 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.934089 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.936158 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:05.936160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:05.936160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:05.936167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:05.937510 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.937993 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.937994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.937994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.938054 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.938285 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.938286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.938286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.938407 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:05.938462 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:05.938749 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 21:13:05.938839 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:13:05.939152 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.930348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34765 (local address 127.0.98.3:60689)
0927 21:13:05.930513 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:05.930552 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:05.931022 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:05.931239 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:05.931242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:05.931919 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:05.932130 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.932136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.932826 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.932826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.933724 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.933727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.934101 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.934102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.934202 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.934746 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:05.934755 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:05.936069 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:05.937597 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:05.937607 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:05.937610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:05.937927 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:05.938110 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:05.938110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:05.938111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:05.938229 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:05.938471 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:05.938478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:05.938620 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 21:13:05.938713 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:13:05.938883 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260927 21:13:05.939602 11083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34765
I20260927 21:13:05.939693 11083 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:05.939937 11083 heartbeater.cc:511] Master 127.0.98.62:34765 requested a full tablet report, sending...
I20260927 21:13:05.940342 10624 ts_manager.cc:194] Registered new tserver with Master: b11f1a397cd649f4b663ce9728e80170 (127.0.98.3:41283)
I20260927 21:13:05.940727 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60689
I20260927 21:13:05.949239 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:13:05.956800 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.950582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.0.98.62:34765)
0927 21:13:05.950643 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:05.950644 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:05.950789 (+ 145us) server_negotiation.cc:407] Connection header received
0927 21:13:05.950819 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:05.950819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:05.950830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:05.950879 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:05.951433 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.951964 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.952794 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.952980 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.953029 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.954296 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:05.954298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:05.954298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:05.954305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:05.955602 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.955990 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.955991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.955992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.956056 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:05.956278 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:05.956279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:05.956279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:05.956434 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:05.956556 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:05.956648 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:13:05.956720 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:13:05.958730 10624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:13:05.960352 10624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:13:05.967127 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.962748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.0.1:43760)
0927 21:13:05.963071 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:05.963079 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:05.963122 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:05.963459 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:05.963460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:05.963462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:05.963499 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.963500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.964975 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.964975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.965993 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.965995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.966850 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.966850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.966919 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.966930 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:13:05.966985 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:13:05.967035 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":34,"thread_start_us":53,"threads_started":1}
I20260927 21:13:05.967136 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.963169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.0.98.1:38451)
0927 21:13:05.963337 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:05.963341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:05.963350 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:13:05.963385 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:05.963387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:05.963401 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:05.963459 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:05.963575 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.964688 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.966071 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.966767 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.966844 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.966907 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:05.967002 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 21:13:05.967048 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260927 21:13:05.967473 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.962851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.0.98.2:34045)
0927 21:13:05.963117 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:05.963122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:05.963132 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:13:05.963172 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:05.963174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:05.963189 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:05.963256 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:05.963404 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.964683 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.966285 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.966985 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.967049 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.967096 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:05.967356 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 21:13:05.967399 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":78,"threads_started":1}
I20260927 21:13:05.967291 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.962740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34045 (local address 127.0.0.1:40732)
0927 21:13:05.962851 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:05.962892 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:05.962959 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:05.963269 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:05.963272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:05.963275 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:05.963349 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.963351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.964794 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.964795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.966197 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.966199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.967075 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.967076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.967133 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.967136 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:13:05.967183 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:13:05.967224 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":149,"thread_start_us":63,"threads_started":1}
I20260927 21:13:05.968150 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.962951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41283 (local address 127.0.0.1:47694)
0927 21:13:05.963227 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:05.963238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:05.963290 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:05.963817 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:05.963817 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:05.963820 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:05.963864 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.963866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.964850 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.964851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.966196 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:05.966199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.967920 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.967920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:05.967981 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.967985 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:13:05.968028 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 21:13:05.968071 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":38,"thread_start_us":62,"threads_started":1}
I20260927 21:13:05.968436 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:05.963061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.0.98.3:41283)
0927 21:13:05.963569 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:05.963574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:05.963582 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:13:05.963616 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:05.963618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:05.963632 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:05.963738 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:05.963922 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.964748 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.967136 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:05.967765 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:05.967829 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:05.967886 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:05.968318 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 21:13:05.968363 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":72,"threads_started":1}
I20260927 21:13:05.968353 10749 tablet_service.cc:1467] Processing CreateTablet for tablet 8a020b7adebe49bc976169a586ed449c (DEFAULT_TABLE table=test-table [id=821c5bbadb2f47d9bbcbe7d6d5af3a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:13:05.968350 10883 tablet_service.cc:1467] Processing CreateTablet for tablet 8a020b7adebe49bc976169a586ed449c (DEFAULT_TABLE table=test-table [id=821c5bbadb2f47d9bbcbe7d6d5af3a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:13:05.968716 10883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a020b7adebe49bc976169a586ed449c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:05.968716 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a020b7adebe49bc976169a586ed449c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:05.969017 11017 tablet_service.cc:1467] Processing CreateTablet for tablet 8a020b7adebe49bc976169a586ed449c (DEFAULT_TABLE table=test-table [id=821c5bbadb2f47d9bbcbe7d6d5af3a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:13:05.969347 11017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a020b7adebe49bc976169a586ed449c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:05.971942 11100 tablet_bootstrap.cc:492] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: Bootstrap starting.
I20260927 21:13:05.972230 11101 tablet_bootstrap.cc:492] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b: Bootstrap starting.
I20260927 21:13:05.972369 11102 tablet_bootstrap.cc:492] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: Bootstrap starting.
I20260927 21:13:05.972693 11100 tablet_bootstrap.cc:654] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:05.972929 11101 tablet_bootstrap.cc:654] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:05.973016 11102 tablet_bootstrap.cc:654] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:05.973189 11100 log.cc:824] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:05.973510 11101 log.cc:824] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:05.973538 11102 log.cc:824] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:05.974030 11100 tablet_bootstrap.cc:492] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: No bootstrap required, opened a new log
I20260927 21:13:05.974119 11100 ts_tablet_manager.cc:1397] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:13:05.974226 11102 tablet_bootstrap.cc:492] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: No bootstrap required, opened a new log
I20260927 21:13:05.974323 11102 ts_tablet_manager.cc:1397] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:13:05.974473 11101 tablet_bootstrap.cc:492] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b: No bootstrap required, opened a new log
I20260927 21:13:05.974537 11101 ts_tablet_manager.cc:1397] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:13:05.975627 11100 raft_consensus.cc:348] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } }
I20260927 21:13:05.975943 11100 raft_consensus.cc:374] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:05.976042 11100 raft_consensus.cc:729] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11f1a397cd649f4b663ce9728e80170, State: Initialized, Role: FOLLOWER
I20260927 21:13:05.976203 11100 consensus_queue.cc:260] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } }
I20260927 21:13:05.976310 11102 raft_consensus.cc:348] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } }
I20260927 21:13:05.976482 11100 ts_tablet_manager.cc:1428] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:13:05.976408 11101 raft_consensus.cc:348] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } }
I20260927 21:13:05.976512 11102 raft_consensus.cc:374] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:05.976552 11102 raft_consensus.cc:729] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165af050b4ab4ae9be5cfc0ac9b4e9fc, State: Initialized, Role: FOLLOWER
I20260927 21:13:05.976598 11101 raft_consensus.cc:374] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:05.976608 11083 heartbeater.cc:503] Master 127.0.98.62:34765 was elected leader, sending a full tablet report...
I20260927 21:13:05.976657 11101 raft_consensus.cc:729] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9401be9f2854fadb018e1cd275b7f2b, State: Initialized, Role: FOLLOWER
I20260927 21:13:05.976660 11102 consensus_queue.cc:260] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } }
I20260927 21:13:05.976770 11101 consensus_queue.cc:260] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } }
I20260927 21:13:05.976936 11102 ts_tablet_manager.cc:1428] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:13:05.977059 10949 heartbeater.cc:503] Master 127.0.98.62:34765 was elected leader, sending a full tablet report...
I20260927 21:13:05.977322 10815 heartbeater.cc:503] Master 127.0.98.62:34765 was elected leader, sending a full tablet report...
I20260927 21:13:05.977730 11101 ts_tablet_manager.cc:1428] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260927 21:13:06.035384 10950 tablet.cc:2369] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:13:06.056382 11106 raft_consensus.cc:482] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:13:06.056529 11106 raft_consensus.cc:504] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } }
I20260927 21:13:06.056932 11106 leader_election.cc:288] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9401be9f2854fadb018e1cd275b7f2b (127.0.98.1:38451), 165af050b4ab4ae9be5cfc0ac9b4e9fc (127.0.98.2:34045)
I20260927 21:13:06.059769 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:06.057160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38383 (local address 127.0.98.1:38451)
0927 21:13:06.057209 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:06.057211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:06.057251 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:13:06.057299 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:06.057300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:06.057311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:06.057355 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:06.057507 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.058091 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.059057 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.059438 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.059489 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:06.059498 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:06.059654 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 21:13:06.059699 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:13:06.059778 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:06.057190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38491 (local address 127.0.98.2:34045)
0927 21:13:06.057229 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:06.057231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:06.057531 (+ 300us) server_negotiation.cc:407] Connection header received
0927 21:13:06.057560 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:06.057561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:06.057572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:06.057614 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:06.057752 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.058216 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.059107 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.059478 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.059544 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:06.059554 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:06.059682 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 21:13:06.059720 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:13:06.059770 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:06.057092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.98.3:38383)
0927 21:13:06.057152 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:06.057164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:06.057203 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:06.057364 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:06.057365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:06.057367 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:06.057438 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:06.057440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.058177 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.058178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:06.058966 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:06.058968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.059510 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.059511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:06.059587 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:06.059591 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:13:06.059634 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 21:13:06.059677 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:13:06.059788 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:06.057092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34045 (local address 127.0.98.3:38491)
0927 21:13:06.057455 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:06.057464 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:06.057521 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:06.057632 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:06.057632 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:06.057634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:06.057694 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:06.057696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.058287 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.058288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:06.059021 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:06.059024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.059560 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.059561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:06.059612 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:06.059615 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:13:06.059665 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:13:06.059698 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":151,"thread_start_us":145,"threads_started":1}
I20260927 21:13:06.060146 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a020b7adebe49bc976169a586ed449c" candidate_uuid: "b11f1a397cd649f4b663ce9728e80170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9401be9f2854fadb018e1cd275b7f2b" is_pre_election: true
I20260927 21:13:06.060146 10903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a020b7adebe49bc976169a586ed449c" candidate_uuid: "b11f1a397cd649f4b663ce9728e80170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" is_pre_election: true
I20260927 21:13:06.060343 10769 raft_consensus.cc:2454] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b11f1a397cd649f4b663ce9728e80170 in term 0.
I20260927 21:13:06.060362 10903 raft_consensus.cc:2454] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b11f1a397cd649f4b663ce9728e80170 in term 0.
I20260927 21:13:06.060567 10972 leader_election.cc:302] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 165af050b4ab4ae9be5cfc0ac9b4e9fc, b11f1a397cd649f4b663ce9728e80170; no voters:
I20260927 21:13:06.060739 11106 raft_consensus.cc:2781] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:13:06.060779 11106 raft_consensus.cc:482] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:13:06.060793 11106 raft_consensus.cc:3037] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:06.061506 11106 raft_consensus.cc:504] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } }
I20260927 21:13:06.061635 11106 leader_election.cc:288] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [CANDIDATE]: Term 1 election: Requested vote from peers c9401be9f2854fadb018e1cd275b7f2b (127.0.98.1:38451), 165af050b4ab4ae9be5cfc0ac9b4e9fc (127.0.98.2:34045)
I20260927 21:13:06.061789 10903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a020b7adebe49bc976169a586ed449c" candidate_uuid: "b11f1a397cd649f4b663ce9728e80170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc"
I20260927 21:13:06.061789 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a020b7adebe49bc976169a586ed449c" candidate_uuid: "b11f1a397cd649f4b663ce9728e80170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9401be9f2854fadb018e1cd275b7f2b"
I20260927 21:13:06.061864 10769 raft_consensus.cc:3037] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:06.061865 10903 raft_consensus.cc:3037] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:06.062728 10769 raft_consensus.cc:2454] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b11f1a397cd649f4b663ce9728e80170 in term 1.
I20260927 21:13:06.062728 10903 raft_consensus.cc:2454] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b11f1a397cd649f4b663ce9728e80170 in term 1.
I20260927 21:13:06.062922 10972 leader_election.cc:302] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 165af050b4ab4ae9be5cfc0ac9b4e9fc, b11f1a397cd649f4b663ce9728e80170; no voters:
I20260927 21:13:06.063002 11106 raft_consensus.cc:2781] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:13:06.063192 11106 raft_consensus.cc:686] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 1 LEADER]: Becoming Leader. State: Replica: b11f1a397cd649f4b663ce9728e80170, State: Running, Role: LEADER
I20260927 21:13:06.063285 11106 consensus_queue.cc:237] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } }
I20260927 21:13:06.064107 10623 catalog_manager.cc:5495] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 reported cstate change: term changed from 0 to 1, leader changed from <none> to b11f1a397cd649f4b663ce9728e80170 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b11f1a397cd649f4b663ce9728e80170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b11f1a397cd649f4b663ce9728e80170" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41283 } health_report { overall_health: HEALTHY } } }
I20260927 21:13:06.085419 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:06.082870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.0.98.3:41283)
0927 21:13:06.082944 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:06.082945 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:06.082952 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:13:06.082984 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:06.082985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:06.083008 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:06.083079 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:13:06.083257 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.083743 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.084508 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:06.084758 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:06.084848 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:06.085039 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:13:06.085162 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:13:06.085237 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:06.085285 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 21:13:06.085313 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:13:06.087600 10769 raft_consensus.cc:1261] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [term 1 FOLLOWER]: Refusing update from remote peer b11f1a397cd649f4b663ce9728e80170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:13:06.087757 10903 raft_consensus.cc:1261] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [term 1 FOLLOWER]: Refusing update from remote peer b11f1a397cd649f4b663ce9728e80170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:13:06.088078 11106 consensus_queue.cc:1035] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "165af050b4ab4ae9be5cfc0ac9b4e9fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:13:06.088294 11110 consensus_queue.cc:1035] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9401be9f2854fadb018e1cd275b7f2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:13:06.091022 11113 mvcc.cc:205] Tried to move back new op lower bound from 7334066528611209216 to 7334066528515956736. Current Snapshot: MvccSnapshot[applied={T|T < 7334066528611209216 or (T in {7334066528611209216})}]
I20260927 21:13:06.091948 11114 mvcc.cc:205] Tried to move back new op lower bound from 7334066528611209216 to 7334066528515956736. Current Snapshot: MvccSnapshot[applied={T|T < 7334066528611209216}]
I20260927 21:13:06.092650 11116 mvcc.cc:205] Tried to move back new op lower bound from 7334066528611209216 to 7334066528515956736. Current Snapshot: MvccSnapshot[applied={T|T < 7334066528611209216}]
I20260927 21:13:06.093364 10624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:13:06.093465 10624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:13:06.094216 10624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9084cfb076494f439fc2164900e69ab7: Sending DeleteTablet for 3 replicas of tablet 8a020b7adebe49bc976169a586ed449c
I20260927 21:13:06.094519 10749 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a020b7adebe49bc976169a586ed449c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:13:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43760
I20260927 21:13:06.094534 11017 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a020b7adebe49bc976169a586ed449c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:13:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47694
I20260927 21:13:06.094614 10883 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a020b7adebe49bc976169a586ed449c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:13:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40732
I20260927 21:13:06.094791 11123 tablet_replica.cc:330] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b: stopping tablet replica
I20260927 21:13:06.094905 11123 raft_consensus.cc:2229] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:13:06.094966 11122 tablet_replica.cc:330] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: stopping tablet replica
I20260927 21:13:06.095070 11122 raft_consensus.cc:2229] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:13:06.095132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 10686
I20260927 21:13:06.095172 11122 raft_consensus.cc:2258] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:13:06.095233 11123 raft_consensus.cc:2258] T 8a020b7adebe49bc976169a586ed449c P c9401be9f2854fadb018e1cd275b7f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:13:06.095631 11122 ts_tablet_manager.cc:1905] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:13:06.095890 11124 tablet_replica.cc:330] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: stopping tablet replica
I20260927 21:13:06.096046 11124 raft_consensus.cc:2229] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:13:06.096208 11124 raft_consensus.cc:2258] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:13:06.096658 11124 ts_tablet_manager.cc:1905] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:13:06.097579 11122 ts_tablet_manager.cc:1918] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:13:06.097661 11122 log.cc:1196] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/8a020b7adebe49bc976169a586ed449c
I20260927 21:13:06.097931 11122 ts_tablet_manager.cc:1939] T 8a020b7adebe49bc976169a586ed449c P b11f1a397cd649f4b663ce9728e80170: Deleting consensus metadata
I20260927 21:13:06.098371 10608 catalog_manager.cc:4841] TS b11f1a397cd649f4b663ce9728e80170 (127.0.98.3:41283): tablet 8a020b7adebe49bc976169a586ed449c (table test-table [id=821c5bbadb2f47d9bbcbe7d6d5af3a3c]) successfully deleted
I20260927 21:13:06.098626 11124 ts_tablet_manager.cc:1918] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:13:06.098704 11124 log.cc:1196] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/8a020b7adebe49bc976169a586ed449c
I20260927 21:13:06.098951 11124 ts_tablet_manager.cc:1939] T 8a020b7adebe49bc976169a586ed449c P 165af050b4ab4ae9be5cfc0ac9b4e9fc: Deleting consensus metadata
I20260927 21:13:06.099382 10610 catalog_manager.cc:4841] TS 165af050b4ab4ae9be5cfc0ac9b4e9fc (127.0.98.2:34045): tablet 8a020b7adebe49bc976169a586ed449c (table test-table [id=821c5bbadb2f47d9bbcbe7d6d5af3a3c]) successfully deleted
W20260927 21:13:06.101871 10608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38451 (error 108)
I20260927 21:13:06.101956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 10820
I20260927 21:13:06.103016 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:06.102886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.0.1:43776)
0927 21:13:06.102939 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:06.102976 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 21:13:06.103168 10608 catalog_manager.cc:4584] TS c9401be9f2854fadb018e1cd275b7f2b (127.0.98.1:38451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a020b7adebe49bc976169a586ed449c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38451: connect: Connection refused (error 111)
I20260927 21:13:06.106832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 10954
I20260927 21:13:06.111250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 10592
2026-09-27T21:13:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1877 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:13:06 dist-test-slave-r9x1 krb5kdc[11128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:13:06 dist-test-slave-r9x1 krb5kdc[11128](info): set up 2 sockets
Sep 27 21:13:06 dist-test-slave-r9x1 krb5kdc[11128](info): commencing operation
krb5kdc: starting...
W20260927 21:13:07.074116 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:13:07 dist-test-slave-r9x1 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:13:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 21:13:07.246374 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskm7AuFZ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/postgres -l logfile start
2026-09-27 21:13:10.009 UTC [11158] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:13:10.009 UTC [11158] LOG: listening on IPv4 address "127.0.98.20", port 41377
2026-09-27 21:13:10.013 UTC [11158] LOG: listening on Unix socket "/tmp/.s.PGSQL.41377"
2026-09-27 21:13:10.035 UTC [11161] LOG: database system was shut down at 2026-09-27 21:13:08 UTC
2026-09-27 21:13:10.038 UTC [11158] LOG: database system is ready to accept connections
I20260927 21:13:10.913817 392 mini_postgres.cc:96] Postgres bound to 41377
2026-09-27 21:13:10.919 UTC [11169] FATAL: database "slave" does not exist
127.0.98.20:41377 - accepting connections
I20260927 21:13:10.920075 392 mini_ranger.cc:161] Starting Ranger...
I20260927 21:13:10.934888 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:13:11.498942 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:13:11.499013 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-admin
2026-09-27 21:13:11,775 [I] DB FLAVOR :POSTGRES
2026-09-27 21:13:11,775 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:13:11,775 [I] Checking connection..
2026-09-27 21:13:11,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:13:12,023 [I] Checking connection passed.
2026-09-27 21:13:12,023 [I] --------- Verifying version history table ---------
2026-09-27 21:13:12,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:13:12,279 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:13:12,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:13:12,518 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:13:12,519 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:13:12,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:13:12.781 UTC [11233] WARNING: there is no transaction in progress
2026-09-27 21:13:12,787 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:13:12,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:13:13,035 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:13:13,036 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:13:13,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:13:13,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:13:13,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r9x1.c.gcp-upstream.internal','N') ;"
2026-09-27 21:13:13,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:13:14,036 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:13:14,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:13:14,288 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:13:14,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:13:14,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:13:14,811 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:13:14,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:13:15.915 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:15.933 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:15.952 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:15.973 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:15.998 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.018 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.085 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.094 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.111 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.123 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.135 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.151 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.163 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.171 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.181 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:16.191 UTC [11341] WARNING: there is no transaction in progress
2026-09-27 21:13:17,013 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:13:17,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:13:17,279 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:13:17,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:13:17,553 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:13:17,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:13:17,818 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:13:17,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:13:18,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:13:18,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r9x1.c.gcp-upstream.internal';"
2026-09-27 21:13:18,585 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:13:18,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:13:18,845 [I] DB_PATCHES have already been applied
I20260927 21:13:18.851109 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskm7AuFZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:13:18.851186 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 21:13:18.854072 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:54863
Sep 27, 2026 9:13:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:13:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:13:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:13:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:13:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:13:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:13:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:13:19.855557 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.935s user 0.000s sys 0.008s
Sep 27, 2026 9:13:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:13:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 9:13:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:13:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:13:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:13:43.568110 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:13:43 dist-test-slave-r9x1 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543623, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:13:43.655602 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:13:43.665637 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:13:44.628616 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:13:44.628715 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-kms
I20260927 21:13:44.628731 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41377
2026-09-27 21:13:44,807 [I] DB FLAVOR :POSTGRES
2026-09-27 21:13:44,807 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:13:44,808 [I] Checking connection
2026-09-27 21:13:44,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:13:45,109 [I] connection success
2026-09-27 21:13:45,110 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:13:45,110 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:13:45,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:13:45,393 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:13:45,393 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:13:45,393 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:13:45,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41377/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:13:45,756 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:13:46.274802 392 mini_ranger_kms.cc:310] Created kms service
I20260927 21:13:46.483318 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:13:46.531786 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:13:46 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543623, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:13:46.925626 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:13:46.925757 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:13:46.925779 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 21:13:46.929546 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11522
I20260927 21:13:46.929682 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:53251
Sep 27, 2026 9:13:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:13:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:13:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:13:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:13:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:13:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 21:13:49.043916 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.388s user 0.028s sys 0.133s
I20260927 21:13:49.044024 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:13:49.044095 392 mini_ranger_kms.cc:401] 127.0.98.20:53251/kms/v1/keys
Sep 27 21:13:49 dist-test-slave-r9x1 krb5kdc[11128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543629, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 21:13:49 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543629, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 21:13:50 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543629, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 21:13:52 dist-test-slave-r9x1 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:13:52.544100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44211
--ranger_config_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53251/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:13:52.629513 11572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:52.629736 11572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:52.629758 11572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:52.631076 11572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:13:52.631136 11572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:52.631150 11572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:52.631161 11572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:13:52.631173 11572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:13:52.632926 11572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:53251/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44211
--ranger_config_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:52.633188 11572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:52.633476 11572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:52.638850 11580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:52.638895 11577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:52.638928 11578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:52.639000 11572 server_base.cc:1034] running on GCE node
I20260927 21:13:52.639235 11572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:52.639530 11572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:52.640693 11572 hybrid_clock.cc:648] HybridClock initialized: now 1790543632640670 us; error 40 us; skew 500 ppm
Sep 27 21:13:52 dist-test-slave-r9x1 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:13:52.643836 11572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:13:52.644320 11572 webserver.cc:460] Webserver started at http://127.0.98.62:41329/ using document root <none> and password file <none>
I20260927 21:13:52.644636 11572 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:52.644703 11572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:52.644829 11572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:13:52 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:13:52.695101 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "c3034c52d44940638411f5f76280a3d7"
format_stamp: "Formatted at 2026-09-27 21:13:52 on dist-test-slave-r9x1"
server_key: "2052f15e03b2f6e3083e12eeb0b3a4ed"
server_key_iv: "f5fa09abf8485a704dc15db622f052e7"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:52.695533 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3034c52d44940638411f5f76280a3d7"
format_stamp: "Formatted at 2026-09-27 21:13:52 on dist-test-slave-r9x1"
server_key: "2052f15e03b2f6e3083e12eeb0b3a4ed"
server_key_iv: "f5fa09abf8485a704dc15db622f052e7"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:52.696942 11572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:13:52.709208 11587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:52.709507 11572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:13:52.709609 11572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "c3034c52d44940638411f5f76280a3d7"
format_stamp: "Formatted at 2026-09-27 21:13:52 on dist-test-slave-r9x1"
server_key: "2052f15e03b2f6e3083e12eeb0b3a4ed"
server_key_iv: "f5fa09abf8485a704dc15db622f052e7"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:52.709728 11572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:52.723203 11572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:52.725175 11572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:52.725358 11572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:52.730360 11572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44211
I20260927 21:13:52.730377 11639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44211 every 8 connection(s)
I20260927 21:13:52.730792 11572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:13:52.731684 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:52.734155 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7: Bootstrap starting.
I20260927 21:13:52.734757 11640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:52.735020 11640 log.cc:824] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:52.735687 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7: No bootstrap required, opened a new log
I20260927 21:13:52.737739 11640 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3034c52d44940638411f5f76280a3d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260927 21:13:52.737886 11640 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:52.737917 11640 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3034c52d44940638411f5f76280a3d7, State: Initialized, Role: FOLLOWER
I20260927 21:13:52.738065 11640 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3034c52d44940638411f5f76280a3d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260927 21:13:52.738132 11640 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:13:52.738157 11640 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:13:52.738200 11640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:52.738754 11640 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3034c52d44940638411f5f76280a3d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260927 21:13:52.738845 11640 leader_election.cc:302] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3034c52d44940638411f5f76280a3d7; no voters:
I20260927 21:13:52.739017 11640 leader_election.cc:288] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:13:52.739054 11643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:13:52.739207 11643 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [term 1 LEADER]: Becoming Leader. State: Replica: c3034c52d44940638411f5f76280a3d7, State: Running, Role: LEADER
I20260927 21:13:52.739289 11640 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:13:52.739341 11643 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3034c52d44940638411f5f76280a3d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260927 21:13:52.739730 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3034c52d44940638411f5f76280a3d7. Latest consensus state: current_term: 1 leader_uuid: "c3034c52d44940638411f5f76280a3d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3034c52d44940638411f5f76280a3d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } } }
I20260927 21:13:52.739742 11645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3034c52d44940638411f5f76280a3d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3034c52d44940638411f5f76280a3d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } } }
I20260927 21:13:52.739926 11645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:13:52.740046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 11572
I20260927 21:13:52.740126 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:13:52.740144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:13:52.740253 11652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:13:52.740324 392 external_mini_cluster.cc:1428] Setting key 0a78db742998dcc9221438c49a998ec7
I20260927 21:13:52.742803 11652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:13:52.743989 11640 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:13:52.748764 11652 catalog_manager.cc:1347] Generated new cluster ID: ac9c7be844ba4d9bbbb0be21fb471839
I20260927 21:13:52.748895 11652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:13:52 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:13:52.767131 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:52.741002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.0.98.62:44211)
0927 21:13:52.741975 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:52.741986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:52.742020 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:13:52.742199 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:52.742208 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:52.742273 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:52.742428 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:52.744024 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:52.745478 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:52.758597 (+ 13119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:52.759333 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:52.759813 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:52.762659 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:52.762666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:52.762667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:52.762683 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:52.764824 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:52.765302 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:52.765306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:52.765306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:52.765432 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:52.765630 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:52.765631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:52.765631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:52.766013 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:52.766156 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:52.766717 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 21:13:52.766870 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":812,"threads_started":1}
I20260927 21:13:52.780665 11652 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:13:52.781589 11652 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:13:52.785250 11652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7: Generated new TSK 0
I20260927 21:13:52.785627 11652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:13:53.573912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:13:53.701325 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:53.701751 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:53.701830 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:53.703753 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:53.703821 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:53.703866 11685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:13:53.706197 11685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:53251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:53.706625 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:53.706925 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:53.714273 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:53.714493 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:53.714998 11685 server_base.cc:1034] running on GCE node
W20260927 21:13:53.715171 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:53.715438 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:53.715765 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:53.716957 11685 hybrid_clock.cc:648] HybridClock initialized: now 1790543633716924 us; error 42 us; skew 500 ppm
Sep 27 21:13:53 dist-test-slave-r9x1 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:13:53.720454 11685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:13:53.721022 11685 webserver.cc:460] Webserver started at http://127.0.98.1:34591/ using document root <none> and password file <none>
I20260927 21:13:53.721316 11685 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:53.721388 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:53.721520 11685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:13:53 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:13:53.739202 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "0169227b1ad84d2a8d1f860bf352a643"
format_stamp: "Formatted at 2026-09-27 21:13:53 on dist-test-slave-r9x1"
server_key: "21a1c98370b0290853ff397ea4f83fba"
server_key_iv: "ac1356b1f05a3bd0efe1e87cf503e06b"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:53.739737 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0169227b1ad84d2a8d1f860bf352a643"
format_stamp: "Formatted at 2026-09-27 21:13:53 on dist-test-slave-r9x1"
server_key: "21a1c98370b0290853ff397ea4f83fba"
server_key_iv: "ac1356b1f05a3bd0efe1e87cf503e06b"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:53.741518 11685 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:13:53.755860 11700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:53.756070 11685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:13:53.756170 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "0169227b1ad84d2a8d1f860bf352a643"
format_stamp: "Formatted at 2026-09-27 21:13:53 on dist-test-slave-r9x1"
server_key: "21a1c98370b0290853ff397ea4f83fba"
server_key_iv: "ac1356b1f05a3bd0efe1e87cf503e06b"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:53.756263 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:53.769469 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:53.774506 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:53.774747 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:53.774991 11685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:53.775434 11685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:53.775480 11685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:53.775507 11685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:53.775552 11685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:53.782971 11685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46131
I20260927 21:13:53.783545 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:13:53.784911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 11685
I20260927 21:13:53.785012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:13:53.785198 392 external_mini_cluster.cc:1428] Setting key 0b8be3a95a9a032279d513548ed21590
I20260927 21:13:53.784631 11815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46131 every 8 connection(s)
Sep 27 21:13:53 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 21:13:53.803035 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:53.788471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38407 (local address 127.0.98.62:44211)
0927 21:13:53.789618 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:53.789620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:53.789628 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:13:53.789652 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:53.789653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:53.789665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:53.789716 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:53.793035 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:53.793896 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:53.795310 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:53.795639 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:53.795727 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:53.798524 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:53.798525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:53.798526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:53.798547 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:53.800365 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:53.801643 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:53.801645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:53.801645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:53.801722 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:53.802114 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:53.802115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:53.802115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:53.802275 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:53.802357 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:53.802827 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 21:13:53.802926 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1123,"thread_start_us":52,"threads_started":1}
I20260927 21:13:53.803130 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:53.788137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44211 (local address 127.0.98.1:38407)
0927 21:13:53.788723 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:53.788772 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:53.789434 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:53.792317 (+ 2883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:53.792323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:53.792594 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:53.792895 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:53.792903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:53.794020 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:53.794022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:53.795174 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:53.795178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:53.795815 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:53.795817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:53.795979 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:53.796663 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:53.796674 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:53.798367 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:53.801089 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:53.801102 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:53.801105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:53.801533 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:53.801819 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:53.801820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:53.801821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:53.801993 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:53.802352 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:53.802359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:53.802571 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 21:13:53.802690 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 21:13:53.802876 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 21:13:53.805459 11816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44211
I20260927 21:13:53.805598 11816 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:53.805835 11816 heartbeater.cc:511] Master 127.0.98.62:44211 requested a full tablet report, sending...
I20260927 21:13:53.806519 11604 ts_manager.cc:194] Registered new tserver with Master: 0169227b1ad84d2a8d1f860bf352a643 (127.0.98.1:46131)
I20260927 21:13:53.807456 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:13:53.828699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:13:53.954945 11822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:53.955327 11822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:53.955420 11822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:53.957613 11822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:53.957760 11822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:53.957839 11822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:13:53.960630 11822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:53251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:53.961136 11822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:53.961489 11822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:53.968135 11830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:53.968576 11829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:53.968912 11832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:53.969570 11822 server_base.cc:1034] running on GCE node
I20260927 21:13:53.969801 11822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:53.970068 11822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:53.971232 11822 hybrid_clock.cc:648] HybridClock initialized: now 1790543633971208 us; error 36 us; skew 500 ppm
Sep 27 21:13:53 dist-test-slave-r9x1 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:13:53.974641 11822 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:13:53.975179 11822 webserver.cc:460] Webserver started at http://127.0.98.2:42453/ using document root <none> and password file <none>
I20260927 21:13:53.975890 11822 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:53.976032 11822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:53.976220 11822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:13:53 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 21:13:53 dist-test-slave-r9x1 krb5kdc[11128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:13:53.998167 11822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "809a7c44816944eeb85e46769c087161"
format_stamp: "Formatted at 2026-09-27 21:13:53 on dist-test-slave-r9x1"
server_key: "68731101cec1f0ff3663b17a265d7132"
server_key_iv: "e3831a803463118f8ef5e86337a44a87"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:53.998657 11822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "809a7c44816944eeb85e46769c087161"
format_stamp: "Formatted at 2026-09-27 21:13:53 on dist-test-slave-r9x1"
server_key: "68731101cec1f0ff3663b17a265d7132"
server_key_iv: "e3831a803463118f8ef5e86337a44a87"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:54.003585 11822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260927 21:13:54.025242 11839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:54.026019 11822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:13:54.026139 11822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "809a7c44816944eeb85e46769c087161"
format_stamp: "Formatted at 2026-09-27 21:13:53 on dist-test-slave-r9x1"
server_key: "68731101cec1f0ff3663b17a265d7132"
server_key_iv: "e3831a803463118f8ef5e86337a44a87"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:54.026244 11822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:54.039800 11822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:54.042658 11822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:54.043017 11822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:54.043354 11822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:54.043838 11822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:54.043937 11822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:54.044013 11822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:54.044071 11822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:54.050595 11822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38589
I20260927 21:13:54.051146 11822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:13:54.052034 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38589 every 8 connection(s)
Sep 27 21:13:54 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:13:54.061094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 11822
I20260927 21:13:54.061196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:13:54.061522 392 external_mini_cluster.cc:1428] Setting key 42593b2be4ebdad51c499b500c775b18
I20260927 21:13:54.065161 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.053068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58721 (local address 127.0.98.62:44211)
0927 21:13:54.053117 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:54.053119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:54.053954 (+ 835us) server_negotiation.cc:407] Connection header received
0927 21:13:54.054786 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:54.054788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:54.054802 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:54.054832 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:54.055730 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.056509 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.058118 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.058425 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.058471 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.061526 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:54.061528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:54.061528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:54.061537 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:54.063338 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:54.063949 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:54.063951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:54.063951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:54.064033 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:54.064348 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:54.064349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:54.064349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:54.064496 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:54.064547 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:54.064922 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 21:13:54.065040 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:13:54.065708 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.052767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44211 (local address 127.0.98.2:58721)
0927 21:13:54.053798 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:54.053848 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:54.054556 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:54.055025 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:54.055030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:54.055303 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:54.055613 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.055621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.056651 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.056652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.058004 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.058008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.058581 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.058581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.058735 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.059374 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:54.059385 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:54.061147 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:54.063453 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:54.063465 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:54.063467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:54.063824 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:54.064114 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:54.064116 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:54.064116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:54.064275 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:54.064578 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:54.064584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:54.064773 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 21:13:54.065284 (+ 511us) client_negotiation.cc:240] Negotiation successful
0927 21:13:54.065470 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"thread_start_us":91,"threads_started":1}
I20260927 21:13:54.066311 11953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44211
I20260927 21:13:54.066416 11953 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:54.066612 11953 heartbeater.cc:511] Master 127.0.98.62:44211 requested a full tablet report, sending...
I20260927 21:13:54.067018 11604 ts_manager.cc:194] Registered new tserver with Master: 809a7c44816944eeb85e46769c087161 (127.0.98.2:38589)
I20260927 21:13:54.067505 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58721
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:13:54.100176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:13:54.235927 11959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:13:54.236230 11959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:13:54.236266 11959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:13:54.238206 11959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:13:54.238281 11959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:13:54.238322 11959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:13:54.240648 11959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:53251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:13:54.240935 11959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:13:54.241230 11959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:13:54.248049 11965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:54.248291 11964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:13:54.248466 11967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:13:54.256503 11959 server_base.cc:1034] running on GCE node
I20260927 21:13:54.256954 11959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:13:54.257345 11959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:13:54.260660 11959 hybrid_clock.cc:648] HybridClock initialized: now 1790543634260623 us; error 46 us; skew 500 ppm
Sep 27 21:13:54 dist-test-slave-r9x1 krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:13:54.264356 11959 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:13:54.264941 11959 webserver.cc:460] Webserver started at http://127.0.98.3:40947/ using document root <none> and password file <none>
I20260927 21:13:54.265208 11959 fs_manager.cc:359] Metadata directory not provided
I20260927 21:13:54.265296 11959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:13:54.265436 11959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:13:54 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:13:54.288553 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "616dd0c7de9149c7a29535d9ba8a15e2"
format_stamp: "Formatted at 2026-09-27 21:13:54 on dist-test-slave-r9x1"
server_key: "679c1d84858c41a2ef76e5851d2cabfb"
server_key_iv: "77052673f7438159701d6caec3ace5f7"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:54.289142 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "616dd0c7de9149c7a29535d9ba8a15e2"
format_stamp: "Formatted at 2026-09-27 21:13:54 on dist-test-slave-r9x1"
server_key: "679c1d84858c41a2ef76e5851d2cabfb"
server_key_iv: "77052673f7438159701d6caec3ace5f7"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:54.293985 11959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260927 21:13:54.311172 11974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:54.311369 11959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 21:13:54.311460 11959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "616dd0c7de9149c7a29535d9ba8a15e2"
format_stamp: "Formatted at 2026-09-27 21:13:54 on dist-test-slave-r9x1"
server_key: "679c1d84858c41a2ef76e5851d2cabfb"
server_key_iv: "77052673f7438159701d6caec3ace5f7"
server_key_version: "kuduclusterkey@0"
I20260927 21:13:54.311568 11959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:13:54.342144 11959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:13:54.344379 11959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:13:54.344525 11959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:13:54.344722 11959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:13:54.345094 11959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:13:54.345144 11959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:54.345172 11959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:13:54.345196 11959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:13:54.353330 11959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33147
I20260927 21:13:54.353591 12087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33147 every 8 connection(s)
I20260927 21:13:54.353765 11959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:13:54.354605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 11959
I20260927 21:13:54.354701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:13:54.354898 392 external_mini_cluster.cc:1428] Setting key 4db637aeafa66b88c55ccfaf370681d1
Sep 27 21:13:54 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:13:54.386209 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.354960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34707 (local address 127.0.98.62:44211)
0927 21:13:54.355004 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:54.355006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:54.367803 (+ 12797us) server_negotiation.cc:407] Connection header received
0927 21:13:54.368629 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:54.368630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:54.368644 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:54.368674 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:54.369644 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.370463 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.372228 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.372609 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.372671 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.382356 (+ 9685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:54.382358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:54.382359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:54.382369 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:54.384266 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:54.384910 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:54.384912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:54.384912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:54.384982 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:54.385337 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:54.385338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:54.385338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:54.385497 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:54.385553 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:54.385992 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 21:13:54.386107 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:13:54.387077 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.354697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44211 (local address 127.0.98.3:34707)
0927 21:13:54.367541 (+ 12844us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:54.367686 (+ 145us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:54.368473 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:54.368890 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:54.368895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:54.369186 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:13:54.369522 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.369534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.370592 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.370593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.372094 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.372098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.372784 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.372785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.372995 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.379630 (+ 6635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:13:54.379645 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:13:54.382187 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:13:54.384421 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:54.384436 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:54.384439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:54.384804 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:54.385070 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:13:54.385071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:13:54.385071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:13:54.385233 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:13:54.385587 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:13:54.385594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:13:54.385800 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 21:13:54.386613 (+ 813us) client_negotiation.cc:240] Negotiation successful
0927 21:13:54.386815 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12768,"thread_start_us":77,"threads_started":1}
I20260927 21:13:54.387802 12088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44211
I20260927 21:13:54.387902 12088 heartbeater.cc:464] Registering TS with master...
I20260927 21:13:54.388116 12088 heartbeater.cc:511] Master 127.0.98.62:44211 requested a full tablet report, sending...
I20260927 21:13:54.388577 11603 ts_manager.cc:194] Registered new tserver with Master: 616dd0c7de9149c7a29535d9ba8a15e2 (127.0.98.3:33147)
I20260927 21:13:54.388970 11603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34707
I20260927 21:13:54.399151 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:13:54.408738 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.400872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.0.98.62:44211)
0927 21:13:54.400916 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:54.400918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:54.401182 (+ 264us) server_negotiation.cc:407] Connection header received
0927 21:13:54.401295 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:54.401296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:54.401307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:54.401335 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:13:54.402062 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.402762 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.403738 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.404005 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.404046 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.405851 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:13:54.405853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:13:54.405853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:13:54.405862 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:13:54.407435 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:54.407877 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:54.407879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:54.407879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:54.407949 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:13:54.408131 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:13:54.408132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:13:54.408132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:13:54.408270 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:13:54.408342 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:54.408551 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 21:13:54.408636 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:13:54.411271 11603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:13:54.413628 11603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:13:54.428983 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.416902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33147 (local address 127.0.0.1:58580)
0927 21:13:54.418928 (+ 2026us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:54.418937 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:54.418966 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:54.420018 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:54.420018 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:54.420020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:54.420078 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.420080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.422805 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.422806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.424089 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.424093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.428732 (+ 4639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.428733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.428807 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.428811 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:13:54.428855 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 21:13:54.428902 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2009,"thread_start_us":76,"threads_started":1}
I20260927 21:13:54.429419 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.417097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46131 (local address 127.0.0.1:49268)
0927 21:13:54.419025 (+ 1928us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:54.419032 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:54.419059 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:54.419660 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:54.419664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:54.419666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:54.420150 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.420151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.424317 (+ 4166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.424318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.425755 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.425759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.427940 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.427942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.428053 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.428070 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 21:13:54.428136 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:13:54.428202 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1917,"thread_start_us":149,"threads_started":1}
I20260927 21:13:54.429999 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.417693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.0.98.3:33147)
0927 21:13:54.419421 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:54.419425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:54.419444 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:13:54.419482 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:54.419483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:54.419505 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:54.419566 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:54.421335 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.422721 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.425895 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.426684 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.426755 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.426831 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:54.429859 (+ 3028us) server_negotiation.cc:299] Negotiation successful
0927 21:13:54.429917 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1692,"thread_start_us":72,"threads_started":1}
I20260927 21:13:54.430346 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.417565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.0.98.1:46131)
0927 21:13:54.419157 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:54.419163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:54.419174 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:13:54.419209 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:54.419211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:54.419227 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:54.419287 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:54.420214 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.424245 (+ 4031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.426871 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.427753 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.427835 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.427897 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:54.430220 (+ 2323us) server_negotiation.cc:299] Negotiation successful
0927 21:13:54.430279 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1555,"thread_start_us":53,"threads_started":1}
I20260927 21:13:54.431188 12022 tablet_service.cc:1467] Processing CreateTablet for tablet 9db496388ad94f7ebfcbd2c74cb61bfa (DEFAULT_TABLE table=test-table [id=b51124aa732546adb40b3e68fd00a999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:13:54.431567 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db496388ad94f7ebfcbd2c74cb61bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:54.436658 11742 tablet_service.cc:1467] Processing CreateTablet for tablet 9db496388ad94f7ebfcbd2c74cb61bfa (DEFAULT_TABLE table=test-table [id=b51124aa732546adb40b3e68fd00a999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:13:54.437074 11742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db496388ad94f7ebfcbd2c74cb61bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:54.443378 12108 tablet_bootstrap.cc:492] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643: Bootstrap starting.
I20260927 21:13:54.445771 12108 tablet_bootstrap.cc:654] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:54.447237 12107 tablet_bootstrap.cc:492] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: Bootstrap starting.
I20260927 21:13:54.448902 12107 tablet_bootstrap.cc:654] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:54.449688 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.416674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38589 (local address 127.0.0.1:52534)
0927 21:13:54.418197 (+ 1523us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:54.418250 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:54.418810 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:54.441792 (+ 22982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:54.441794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:54.441797 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:54.441864 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.441867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.443681 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.443683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.444871 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.444876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.449379 (+ 4503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.449381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.449503 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.449511 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:13:54.449546 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 21:13:54.449605 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1489,"thread_start_us":83,"threads_started":1}
I20260927 21:13:54.450224 12108 log.cc:824] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:54.450667 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.416908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.0.98.2:38589)
0927 21:13:54.441605 (+ 24697us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:54.441611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:54.441624 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:13:54.441665 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:54.441667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:54.441685 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:54.441948 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:54.442025 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.443150 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.446035 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.446977 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.450384 (+ 3407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.450507 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:54.450541 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 21:13:54.450592 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24654,"thread_start_us":78,"threads_started":1}
I20260927 21:13:54.451715 12107 log.cc:824] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:54.451865 11887 tablet_service.cc:1467] Processing CreateTablet for tablet 9db496388ad94f7ebfcbd2c74cb61bfa (DEFAULT_TABLE table=test-table [id=b51124aa732546adb40b3e68fd00a999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:13:54.452273 11887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db496388ad94f7ebfcbd2c74cb61bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:13:54.452765 12108 tablet_bootstrap.cc:492] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643: No bootstrap required, opened a new log
I20260927 21:13:54.452886 12108 ts_tablet_manager.cc:1397] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260927 21:13:54.456202 12108 raft_consensus.cc:348] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260927 21:13:54.456561 12108 raft_consensus.cc:374] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:54.456737 12108 raft_consensus.cc:729] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0169227b1ad84d2a8d1f860bf352a643, State: Initialized, Role: FOLLOWER
I20260927 21:13:54.456956 12108 consensus_queue.cc:260] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260927 21:13:54.457371 12108 ts_tablet_manager.cc:1428] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260927 21:13:54.457536 11816 heartbeater.cc:503] Master 127.0.98.62:44211 was elected leader, sending a full tablet report...
I20260927 21:13:54.459220 12107 tablet_bootstrap.cc:492] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: No bootstrap required, opened a new log
I20260927 21:13:54.459383 12107 ts_tablet_manager.cc:1397] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260927 21:13:54.461850 12107 raft_consensus.cc:348] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260927 21:13:54.462149 12107 raft_consensus.cc:374] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:54.462225 12107 raft_consensus.cc:729] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616dd0c7de9149c7a29535d9ba8a15e2, State: Initialized, Role: FOLLOWER
I20260927 21:13:54.462425 12107 consensus_queue.cc:260] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260927 21:13:54.462782 12107 ts_tablet_manager.cc:1428] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:13:54.462947 12088 heartbeater.cc:503] Master 127.0.98.62:44211 was elected leader, sending a full tablet report...
I20260927 21:13:54.466528 12111 tablet_bootstrap.cc:492] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: Bootstrap starting.
I20260927 21:13:54.467729 12111 tablet_bootstrap.cc:654] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: Neither blocks nor log segments found. Creating new log.
I20260927 21:13:54.468068 12111 log.cc:824] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: Log is configured to *not* fsync() on all Append() calls
I20260927 21:13:54.468910 12111 tablet_bootstrap.cc:492] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: No bootstrap required, opened a new log
I20260927 21:13:54.469048 12111 ts_tablet_manager.cc:1397] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:13:54.471628 12111 raft_consensus.cc:348] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260927 21:13:54.471905 12111 raft_consensus.cc:374] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:13:54.471993 12111 raft_consensus.cc:729] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809a7c44816944eeb85e46769c087161, State: Initialized, Role: FOLLOWER
I20260927 21:13:54.472204 12111 consensus_queue.cc:260] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260927 21:13:54.472569 12111 ts_tablet_manager.cc:1428] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:13:54.472764 11953 heartbeater.cc:503] Master 127.0.98.62:44211 was elected leader, sending a full tablet report...
W20260927 21:13:54.539809 11817 tablet.cc:2369] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:13:54.554061 11954 tablet.cc:2369] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:13:54 dist-test-slave-r9x1 krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 21:13:54.604439 12089 tablet.cc:2369] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:13:54.635946 12112 raft_consensus.cc:482] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:13:54.636111 12112 raft_consensus.cc:504] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260927 21:13:54.636499 12112 leader_election.cc:288] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 809a7c44816944eeb85e46769c087161 (127.0.98.2:38589), 616dd0c7de9149c7a29535d9ba8a15e2 (127.0.98.3:33147)
I20260927 21:13:54.653018 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.636873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57101 (local address 127.0.98.2:38589)
0927 21:13:54.637260 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:54.637284 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:54.645502 (+ 8218us) server_negotiation.cc:407] Connection header received
0927 21:13:54.645633 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:54.645635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:54.645649 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:54.645680 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:54.646292 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.646981 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.650976 (+ 3995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.651706 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.651762 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.651774 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:54.652876 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0927 21:13:54.652933 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260927 21:13:54.653370 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.636815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38589 (local address 127.0.98.1:57101)
0927 21:13:54.645402 (+ 8587us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:54.645420 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:54.645547 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:54.645954 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:54.645956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:54.645958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:54.646066 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.646069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.647112 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.647113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.648404 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.648409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.651843 (+ 3434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.651844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.651924 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.651929 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:13:54.653191 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0927 21:13:54.653250 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8555,"thread_start_us":108,"threads_started":1}
I20260927 21:13:54.653751 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.637376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41487 (local address 127.0.98.3:33147)
0927 21:13:54.637417 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:54.637418 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:54.646245 (+ 8827us) server_negotiation.cc:407] Connection header received
0927 21:13:54.648604 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:54.648606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:54.648619 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:54.648648 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:13:54.648870 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.649588 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.652017 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.652574 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.652624 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.652636 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:54.653631 (+ 995us) server_negotiation.cc:299] Negotiation successful
0927 21:13:54.653689 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:13:54.653724 11907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db496388ad94f7ebfcbd2c74cb61bfa" candidate_uuid: "0169227b1ad84d2a8d1f860bf352a643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809a7c44816944eeb85e46769c087161" is_pre_election: true
I20260927 21:13:54.653915 11907 raft_consensus.cc:2454] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0169227b1ad84d2a8d1f860bf352a643 in term 0.
I20260927 21:13:54.654019 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.637032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33147 (local address 127.0.98.1:41487)
0927 21:13:54.646188 (+ 9156us) negotiation.cc:107] Waiting for socket to connect
0927 21:13:54.646199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:13:54.648512 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:13:54.648709 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:13:54.648710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:13:54.648712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:13:54.648789 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.648791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.649728 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.649729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.650849 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:13:54.650853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.652701 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.652702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:13:54.652787 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.652791 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:13:54.653878 (+ 1087us) client_negotiation.cc:240] Negotiation successful
0927 21:13:54.653929 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9136,"thread_start_us":59,"threads_started":1}
I20260927 21:13:54.654222 11703 leader_election.cc:302] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0169227b1ad84d2a8d1f860bf352a643, 809a7c44816944eeb85e46769c087161; no voters:
I20260927 21:13:54.654439 12112 raft_consensus.cc:2781] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:13:54.654513 12112 raft_consensus.cc:482] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:13:54.654542 12112 raft_consensus.cc:3037] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:54.654544 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db496388ad94f7ebfcbd2c74cb61bfa" candidate_uuid: "0169227b1ad84d2a8d1f860bf352a643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" is_pre_election: true
I20260927 21:13:54.654716 12035 raft_consensus.cc:2454] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0169227b1ad84d2a8d1f860bf352a643 in term 0.
I20260927 21:13:54.655516 12112 raft_consensus.cc:504] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260927 21:13:54.655678 12112 leader_election.cc:288] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [CANDIDATE]: Term 1 election: Requested vote from peers 809a7c44816944eeb85e46769c087161 (127.0.98.2:38589), 616dd0c7de9149c7a29535d9ba8a15e2 (127.0.98.3:33147)
I20260927 21:13:54.655901 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db496388ad94f7ebfcbd2c74cb61bfa" candidate_uuid: "0169227b1ad84d2a8d1f860bf352a643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616dd0c7de9149c7a29535d9ba8a15e2"
I20260927 21:13:54.655894 11907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db496388ad94f7ebfcbd2c74cb61bfa" candidate_uuid: "0169227b1ad84d2a8d1f860bf352a643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809a7c44816944eeb85e46769c087161"
I20260927 21:13:54.655992 12035 raft_consensus.cc:3037] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:54.655994 11907 raft_consensus.cc:3037] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:13:54.656910 12035 raft_consensus.cc:2454] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0169227b1ad84d2a8d1f860bf352a643 in term 1.
I20260927 21:13:54.656936 11907 raft_consensus.cc:2454] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0169227b1ad84d2a8d1f860bf352a643 in term 1.
I20260927 21:13:54.657119 11703 leader_election.cc:302] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0169227b1ad84d2a8d1f860bf352a643, 809a7c44816944eeb85e46769c087161; no voters:
I20260927 21:13:54.657241 12112 raft_consensus.cc:2781] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:13:54.657433 12112 raft_consensus.cc:686] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [term 1 LEADER]: Becoming Leader. State: Replica: 0169227b1ad84d2a8d1f860bf352a643, State: Running, Role: LEADER
I20260927 21:13:54.657548 12112 consensus_queue.cc:237] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260927 21:13:54.658452 11603 catalog_manager.cc:5495] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0169227b1ad84d2a8d1f860bf352a643 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0169227b1ad84d2a8d1f860bf352a643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0169227b1ad84d2a8d1f860bf352a643" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260927 21:13:54.708822 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:13:54.705473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.0.98.1:46131)
0927 21:13:54.705577 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 21:13:54.705579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:13:54.705585 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:13:54.705621 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:13:54.705622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:13:54.705643 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:13:54.705699 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:13:54.705860 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.706571 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.707653 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:13:54.707932 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:13:54.707996 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:13:54.708220 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:13:54.708368 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:13:54.708445 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:13:54.708704 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 21:13:54.708728 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 21:13:54.711233 12035 raft_consensus.cc:1261] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [term 1 FOLLOWER]: Refusing update from remote peer 0169227b1ad84d2a8d1f860bf352a643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:13:54.711719 12118 consensus_queue.cc:1035] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616dd0c7de9149c7a29535d9ba8a15e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:13:54.713838 11907 raft_consensus.cc:1261] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [term 1 FOLLOWER]: Refusing update from remote peer 0169227b1ad84d2a8d1f860bf352a643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:13:54.714223 12118 consensus_queue.cc:1035] T 9db496388ad94f7ebfcbd2c74cb61bfa P 0169227b1ad84d2a8d1f860bf352a643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "809a7c44816944eeb85e46769c087161" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:13:54.718729 12121 mvcc.cc:205] Tried to move back new op lower bound from 7334066727774040064 to 7334066727558012928. Current Snapshot: MvccSnapshot[applied={T|T < 7334066727774040064}]
I20260927 21:13:54.719277 12124 mvcc.cc:205] Tried to move back new op lower bound from 7334066727774040064 to 7334066727558012928. Current Snapshot: MvccSnapshot[applied={T|T < 7334066727774040064}]
I20260927 21:13:54.723963 12122 mvcc.cc:205] Tried to move back new op lower bound from 7334066727774040064 to 7334066727558012928. Current Snapshot: MvccSnapshot[applied={T|T < 7334066727774040064}]
I20260927 21:13:54.728291 11603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:13:54.728442 11603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:13:54.729228 11603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3034c52d44940638411f5f76280a3d7: Sending DeleteTablet for 3 replicas of tablet 9db496388ad94f7ebfcbd2c74cb61bfa
I20260927 21:13:54.729806 11887 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db496388ad94f7ebfcbd2c74cb61bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:13:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52534
I20260927 21:13:54.730105 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db496388ad94f7ebfcbd2c74cb61bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:13:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58580
I20260927 21:13:54.732434 11742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db496388ad94f7ebfcbd2c74cb61bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:13:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49268
I20260927 21:13:54.733115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 11685
I20260927 21:13:54.738788 12130 tablet_replica.cc:330] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: stopping tablet replica
I20260927 21:13:54.738976 12130 raft_consensus.cc:2229] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:13:54.739121 12130 raft_consensus.cc:2258] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:13:54.739326 12131 tablet_replica.cc:330] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: stopping tablet replica
I20260927 21:13:54.739450 12131 raft_consensus.cc:2229] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:13:54.739573 12131 raft_consensus.cc:2258] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:13:54.740202 12131 ts_tablet_manager.cc:1905] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:13:54.741041 12130 ts_tablet_manager.cc:1905] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:13:54.742784 12130 ts_tablet_manager.cc:1918] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:13:54.743330 12131 ts_tablet_manager.cc:1918] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:13:54.743451 12131 log.cc:1196] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/9db496388ad94f7ebfcbd2c74cb61bfa
I20260927 21:13:54.744025 12131 ts_tablet_manager.cc:1939] T 9db496388ad94f7ebfcbd2c74cb61bfa P 616dd0c7de9149c7a29535d9ba8a15e2: Deleting consensus metadata
I20260927 21:13:54.744607 11590 catalog_manager.cc:4841] TS 616dd0c7de9149c7a29535d9ba8a15e2 (127.0.98.3:33147): tablet 9db496388ad94f7ebfcbd2c74cb61bfa (table test-table [id=b51124aa732546adb40b3e68fd00a999]) successfully deleted
I20260927 21:13:54.745101 12130 log.cc:1196] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/9db496388ad94f7ebfcbd2c74cb61bfa
I20260927 21:13:54.745481 12130 ts_tablet_manager.cc:1939] T 9db496388ad94f7ebfcbd2c74cb61bfa P 809a7c44816944eeb85e46769c087161: Deleting consensus metadata
W20260927 21:13:54.745920 11588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46131 (error 108)
I20260927 21:13:54.746582 11590 catalog_manager.cc:4841] TS 809a7c44816944eeb85e46769c087161 (127.0.98.2:38589): tablet 9db496388ad94f7ebfcbd2c74cb61bfa (table test-table [id=b51124aa732546adb40b3e68fd00a999]) successfully deleted
I20260927 21:13:54.746851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 11822
I20260927 21:13:54.754977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 11959
I20260927 21:13:54.760263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 11572
I20260927 21:13:54.765934 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:13:55.111191 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:13:55.111299 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:13:55.446347 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:13:55.446 UTC [11158] LOG: received smart shutdown request
2026-09-27 21:13:55.450 UTC [11158] LOG: background worker "logical replication launcher" (PID 11167) exited with exit code 1
2026-09-27 21:13:55.450 UTC [11162] LOG: shutting down
2026-09-27 21:13:55.624 UTC [11158] LOG: database system is shut down
2026-09-27T21:13:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49599 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:13:55 dist-test-slave-r9x1 krb5kdc[12141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:13:55 dist-test-slave-r9x1 krb5kdc[12141](info): set up 2 sockets
Sep 27 21:13:55 dist-test-slave-r9x1 krb5kdc[12141](info): commencing operation
krb5kdc: starting...
W20260927 21:13:56.669066 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:13:56 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:13:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 21:13:56.835776 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskm7AuFZ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/postgres -l logfile start
2026-09-27 21:13:59.474 UTC [12171] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:13:59.474 UTC [12171] LOG: listening on IPv4 address "127.0.98.20", port 59139
2026-09-27 21:13:59.478 UTC [12171] LOG: listening on Unix socket "/tmp/.s.PGSQL.59139"
2026-09-27 21:13:59.498 UTC [12174] LOG: database system was shut down at 2026-09-27 21:13:57 UTC
2026-09-27 21:13:59.501 UTC [12171] LOG: database system is ready to accept connections
I20260927 21:14:00.373486 392 mini_postgres.cc:96] Postgres bound to 59139
2026-09-27 21:14:00.380 UTC [12182] FATAL: database "slave" does not exist
127.0.98.20:59139 - accepting connections
I20260927 21:14:00.381119 392 mini_ranger.cc:161] Starting Ranger...
I20260927 21:14:00.396632 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:14:00.954753 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:14:00.954846 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-admin
2026-09-27 21:14:01,235 [I] DB FLAVOR :POSTGRES
2026-09-27 21:14:01,235 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:14:01,236 [I] Checking connection..
2026-09-27 21:14:01,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:14:01,519 [I] Checking connection passed.
2026-09-27 21:14:01,519 [I] --------- Verifying version history table ---------
2026-09-27 21:14:01,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:14:01,798 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:14:01,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:14:02,068 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:14:02,068 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:14:02,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:14:02.318 UTC [12246] WARNING: there is no transaction in progress
2026-09-27 21:14:02,325 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:14:02,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:14:02,587 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:14:02,588 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:14:02,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:14:02,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:14:03,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r9x1.c.gcp-upstream.internal','N') ;"
2026-09-27 21:14:03,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:14:03,636 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:14:03,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:14:03,893 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:14:03,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:14:04,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:14:04,401 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:14:04,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:14:05.504 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.524 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.543 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.561 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.582 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.603 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.675 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.683 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.701 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.713 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.725 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.741 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.752 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.759 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.767 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:05.775 UTC [12354] WARNING: there is no transaction in progress
2026-09-27 21:14:06,546 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:14:06,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:14:06,808 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:14:06,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:14:07,072 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:14:07,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:14:07,347 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:14:07,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:14:07,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:14:07,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r9x1.c.gcp-upstream.internal';"
2026-09-27 21:14:08,138 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:14:08,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:14:08,386 [I] DB_PATCHES have already been applied
I20260927 21:14:08.391498 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskm7AuFZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:14:08.391578 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 21:14:08.394665 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:56771
Sep 27, 2026 9:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:14:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:14:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:14:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:14:09.379081 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.998s user 0.003s sys 0.005s
Sep 27, 2026 9:14:30 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:14:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 9:14:30 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:14:30 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:14:31 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:14:32.593045 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:14:32 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543672, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:14:32.678653 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:14:32.687835 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:14:33.654651 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:14:33.654906 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-kms
I20260927 21:14:33.654958 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:59139
2026-09-27 21:14:33,809 [I] DB FLAVOR :POSTGRES
2026-09-27 21:14:33,810 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:14:33,810 [I] Checking connection
2026-09-27 21:14:33,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:14:34,050 [I] connection success
2026-09-27 21:14:34,050 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:14:34,050 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:14:34,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:14:34,317 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:14:34,317 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:14:34,317 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:14:34,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59139/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:14:34,590 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:14:35.088331 392 mini_ranger_kms.cc:310] Created kms service
I20260927 21:14:35.313113 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:14:35.362696 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:14:35 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543672, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:35.783720 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:14:35.783960 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskm7AuFZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskm7AuFZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:14:35.783982 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 21:14:35.787115 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12535
I20260927 21:14:35.787197 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:44139
Sep 27, 2026 9:14:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:14:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:14:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:14:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:14:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:14:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 21:14:37.812032 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.133s user 0.009s sys 0.128s
I20260927 21:14:37.812158 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:14:37.812206 392 mini_ranger_kms.cc:401] 127.0.98.20:44139/kms/v1/keys
Sep 27 21:14:38 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543678, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 21:14:38 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543678, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 21:14:38 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543678, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 21:14:41 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:14:41.244735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41133
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41133
--ranger_config_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44139/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:41.328603 12585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:41.328835 12585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:41.328857 12585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:41.330153 12585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:14:41.330219 12585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:41.330231 12585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:41.330242 12585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:14:41.330253 12585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:14:41.331974 12585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:44139/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41133
--ranger_config_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:41.332238 12585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:41.332465 12585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:14:41.337733 12585 server_base.cc:1034] running on GCE node
W20260927 21:14:41.337733 12593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:41.337921 12591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:41.337771 12590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:41.338423 12585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:41.338735 12585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:41.339876 12585 hybrid_clock.cc:648] HybridClock initialized: now 1790543681339858 us; error 38 us; skew 500 ppm
Sep 27 21:14:41 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:41.342584 12585 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:14:41.343086 12585 webserver.cc:460] Webserver started at http://127.0.98.62:45821/ using document root <none> and password file <none>
I20260927 21:14:41.343290 12585 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:41.343331 12585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:41.343457 12585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:14:41 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:41.393570 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "2a2a53fb11874fa599caf70a65be1328"
format_stamp: "Formatted at 2026-09-27 21:14:41 on dist-test-slave-r9x1"
server_key: "a9910c0be42aa34b17cf63969bfa8df4"
server_key_iv: "dbcdc5db5783f97db7d83751ccc7ac4b"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:41.394011 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a2a53fb11874fa599caf70a65be1328"
format_stamp: "Formatted at 2026-09-27 21:14:41 on dist-test-slave-r9x1"
server_key: "a9910c0be42aa34b17cf63969bfa8df4"
server_key_iv: "dbcdc5db5783f97db7d83751ccc7ac4b"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:41.395599 12585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:14:41.409152 12600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:41.409384 12585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:14:41.409478 12585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "2a2a53fb11874fa599caf70a65be1328"
format_stamp: "Formatted at 2026-09-27 21:14:41 on dist-test-slave-r9x1"
server_key: "a9910c0be42aa34b17cf63969bfa8df4"
server_key_iv: "dbcdc5db5783f97db7d83751ccc7ac4b"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:41.409576 12585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:41.425613 12585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:41.427721 12585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:41.427896 12585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:41.432461 12585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41133
I20260927 21:14:41.432521 12652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41133 every 8 connection(s)
I20260927 21:14:41.432884 12585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:14:41.433600 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:41.435967 12653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Bootstrap starting.
I20260927 21:14:41.436604 12653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:41.436892 12653 log.cc:824] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:41.437695 12653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: No bootstrap required, opened a new log
I20260927 21:14:41.439855 12653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260927 21:14:41.440027 12653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:41.440050 12653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2a53fb11874fa599caf70a65be1328, State: Initialized, Role: FOLLOWER
I20260927 21:14:41.440193 12653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260927 21:14:41.440266 12653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:14:41.440294 12653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:14:41.440340 12653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:41.440750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 12585
I20260927 21:14:41.440856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:14:41.441039 392 external_mini_cluster.cc:1428] Setting key 83bb2621ce0089613de549bcb1d0a7de
I20260927 21:14:41.441030 12653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260927 21:14:41.441141 12653 leader_election.cc:302] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a2a53fb11874fa599caf70a65be1328; no voters:
I20260927 21:14:41.441392 12653 leader_election.cc:288] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:14:41.441444 12656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:14:41.441758 12653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:14:41.441916 12656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 1 LEADER]: Becoming Leader. State: Replica: 2a2a53fb11874fa599caf70a65be1328, State: Running, Role: LEADER
I20260927 21:14:41.442065 12656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260927 21:14:41.442423 12659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a2a53fb11874fa599caf70a65be1328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } } }
I20260927 21:14:41.442489 12659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:41.442600 12659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a2a53fb11874fa599caf70a65be1328. Latest consensus state: current_term: 1 leader_uuid: "2a2a53fb11874fa599caf70a65be1328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } } }
I20260927 21:14:41.442638 12659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:41.443197 12653 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:14:41.446113 12667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:14:41.448643 12667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:14:41.450855 12667 catalog_manager.cc:1347] Generated new cluster ID: 28bdcfc201174016b38a2d98871b5f9b
I20260927 21:14:41.451033 12667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:14:41 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:41.457721 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:41.441773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.0.98.62:41133)
0927 21:14:41.442856 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:41.442867 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:41.442905 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:14:41.443028 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:41.443036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:41.443094 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:41.443318 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:41.445632 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:41.448326 (+ 2694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:41.449517 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:41.450168 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:41.450279 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:41.453233 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:41.453239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:41.453239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:41.453255 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:41.455245 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:41.455727 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:41.455731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:41.455731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:41.455850 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:41.456064 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:41.456065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:41.456065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:41.456388 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:41.456525 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:41.457014 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 21:14:41.457163 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":53,"threads_started":1}
I20260927 21:14:41.460156 12667 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:14:41.461011 12667 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:14:41.470465 12667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Generated new TSK 0
I20260927 21:14:41.470825 12667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:14:42.236907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:42.349910 12698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:42.350144 12698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:42.350180 12698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:42.351533 12698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:42.351588 12698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:42.351629 12698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:14:42.353412 12698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:44139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:42.353710 12698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:42.353945 12698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:42.359306 12704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:42.359781 12703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:42.359952 12706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:42.360220 12698 server_base.cc:1034] running on GCE node
I20260927 21:14:42.360662 12698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:42.360951 12698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:42.362214 12698 hybrid_clock.cc:648] HybridClock initialized: now 1790543682362186 us; error 37 us; skew 500 ppm
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:42.365234 12698 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:14:42.365870 12698 webserver.cc:460] Webserver started at http://127.0.98.1:46739/ using document root <none> and password file <none>
I20260927 21:14:42.366173 12698 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:42.366243 12698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:42.366433 12698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:42.382634 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "aaeb6e5a8540439a86c00d0b9125fe97"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "2030eebc6527a6849aa0ecfbe3350944"
server_key_iv: "4234548ed6a65869a9da865c100ab015"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:42.383092 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aaeb6e5a8540439a86c00d0b9125fe97"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "2030eebc6527a6849aa0ecfbe3350944"
server_key_iv: "4234548ed6a65869a9da865c100ab015"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:42.385099 12698 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:14:42.402027 12714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:42.402252 12698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:14:42.402354 12698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "aaeb6e5a8540439a86c00d0b9125fe97"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "2030eebc6527a6849aa0ecfbe3350944"
server_key_iv: "4234548ed6a65869a9da865c100ab015"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:42.402454 12698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:42.420588 12698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:42.422618 12698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:42.422761 12698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:42.422996 12698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:42.423365 12698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:42.423401 12698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:42.423432 12698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:42.423471 12698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:42.430238 12698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39035
I20260927 21:14:42.430675 12698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:14:42.431046 12828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39035 every 8 connection(s)
I20260927 21:14:42.435995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 12698
I20260927 21:14:42.436107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:14:42.436306 392 external_mini_cluster.cc:1428] Setting key 0a1ac4964f0d8caeb08ac6d1c91f236e
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:42.444347 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:42.431473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39787 (local address 127.0.98.62:41133)
0927 21:14:42.431625 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:42.431627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:42.431915 (+ 288us) server_negotiation.cc:407] Connection header received
0927 21:14:42.432580 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:42.432581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:42.432606 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:42.432707 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:42.433445 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.434234 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.435800 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.436132 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.436208 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:42.440542 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:42.440544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:42.440545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:42.440554 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:42.442476 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:42.443053 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:42.443054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:42.443055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:42.443139 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:42.443456 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:42.443458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:42.443458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:42.443609 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:42.443658 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:42.444126 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 21:14:42.444229 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260927 21:14:42.445199 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:42.431438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41133 (local address 127.0.98.1:39787)
0927 21:14:42.431794 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:42.431828 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:42.432450 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:42.432787 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:42.432792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:42.433043 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:42.433350 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:42.433357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.434357 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.434358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:42.435687 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:42.435691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.436300 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.436301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:42.436577 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:42.438693 (+ 2116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:42.438706 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:42.440401 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:42.442598 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:42.442609 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:42.442612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:42.442958 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:42.443213 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:42.443214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:42.443214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:42.443382 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:42.443707 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:42.443715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:42.443919 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 21:14:42.444744 (+ 825us) client_negotiation.cc:240] Negotiation successful
0927 21:14:42.444934 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":79,"threads_started":1}
I20260927 21:14:42.446087 12829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41133
I20260927 21:14:42.446184 12829 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:42.446375 12829 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260927 21:14:42.446974 12617 ts_manager.cc:194] Registered new tserver with Master: aaeb6e5a8540439a86c00d0b9125fe97 (127.0.98.1:39035)
I20260927 21:14:42.447882 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39787
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:14:42.469794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:14:42.595315 12835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:42.595713 12835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:42.595805 12835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:42.597990 12835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:42.598148 12835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:42.598277 12835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:14:42.601289 12835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:44139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:42.601832 12835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:42.602190 12835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:42.609017 12843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:42.609521 12845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:42.611505 12842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:42.612504 12835 server_base.cc:1034] running on GCE node
I20260927 21:14:42.612890 12835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:42.613209 12835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:42.614431 12835 hybrid_clock.cc:648] HybridClock initialized: now 1790543682614413 us; error 44 us; skew 500 ppm
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:42.617456 12835 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:14:42.617980 12835 webserver.cc:460] Webserver started at http://127.0.98.2:36837/ using document root <none> and password file <none>
I20260927 21:14:42.618271 12835 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:42.618328 12835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:42.618464 12835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:42.637331 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "941c38da39b94b5190cf875526949651"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "81fd4ab174fd32ab8b50501dd653f648"
server_key_iv: "04e74d19367a0c3fbb29b104a3525e27"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:42.637898 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "941c38da39b94b5190cf875526949651"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "81fd4ab174fd32ab8b50501dd653f648"
server_key_iv: "04e74d19367a0c3fbb29b104a3525e27"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:42.640153 12835 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:14:42.657974 12852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:42.658219 12835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 21:14:42.658324 12835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "941c38da39b94b5190cf875526949651"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "81fd4ab174fd32ab8b50501dd653f648"
server_key_iv: "04e74d19367a0c3fbb29b104a3525e27"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:42.658443 12835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:42.670360 12835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:42.672896 12835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:42.673050 12835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:42.673306 12835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:42.673765 12835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:42.673815 12835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:42.673856 12835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:42.673873 12835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:42.681684 12835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38863
I20260927 21:14:42.681757 12966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38863 every 8 connection(s)
I20260927 21:14:42.682264 12835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:42.691579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 12835
I20260927 21:14:42.691697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:14:42.691902 392 external_mini_cluster.cc:1428] Setting key abd7609b5ed71881a17a7a37fc79dc62
I20260927 21:14:42.694146 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:42.683431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41225 (local address 127.0.98.62:41133)
0927 21:14:42.683516 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:42.683518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:42.683530 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:14:42.684133 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:42.684135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:42.684147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:42.684235 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:42.685170 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.686000 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.687631 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.687961 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.688047 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:42.690837 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:42.690839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:42.690840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:42.690851 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:42.692436 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:42.692896 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:42.692898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:42.692898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:42.692957 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:42.693327 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:42.693328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:42.693328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:42.693477 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:42.693545 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:42.693943 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 21:14:42.694045 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:14:42.694936 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:42.683063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41133 (local address 127.0.98.2:41225)
0927 21:14:42.683256 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:42.683314 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:42.684005 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:42.684398 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:42.684403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:42.684685 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:42.684994 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:42.685001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.686144 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.686146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:42.687505 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:42.687509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.688087 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.688088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:42.688263 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:42.688933 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:42.688944 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:42.690695 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:42.692529 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:42.692540 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:42.692544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:42.692827 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:42.693015 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:42.693015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:42.693016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:42.693143 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:42.693575 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:42.693583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:42.693784 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 21:14:42.694505 (+ 721us) client_negotiation.cc:240] Negotiation successful
0927 21:14:42.694690 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260927 21:14:42.695389 12967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41133
I20260927 21:14:42.695492 12967 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:42.695683 12967 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260927 21:14:42.696046 12617 ts_manager.cc:194] Registered new tserver with Master: 941c38da39b94b5190cf875526949651 (127.0.98.2:38863)
I20260927 21:14:42.696518 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41225
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:14:42.721771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:42.813742 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:42.813973 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:42.814007 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:42.815392 12972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:42.815452 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:42.815493 12972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:14:42.817217 12972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:44139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:42.817516 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:42.817771 12972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:42.823695 12980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:42.824007 12977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:42.824064 12978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:42.824635 12972 server_base.cc:1034] running on GCE node
I20260927 21:14:42.825062 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:42.825416 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:42.826601 12972 hybrid_clock.cc:648] HybridClock initialized: now 1790543682826575 us; error 43 us; skew 500 ppm
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:42.830294 12972 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:14:42.830883 12972 webserver.cc:460] Webserver started at http://127.0.98.3:43607/ using document root <none> and password file <none>
I20260927 21:14:42.831166 12972 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:42.831225 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:42.831395 12972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:42.861477 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "285c97d7fe1f40dd8595ebd41a2df6da"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "ef8d1e5946624c2341d34349d4efa0ec"
server_key_iv: "1fae67663a80912671d237ab2f0ead3c"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:42.861927 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "285c97d7fe1f40dd8595ebd41a2df6da"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "ef8d1e5946624c2341d34349d4efa0ec"
server_key_iv: "1fae67663a80912671d237ab2f0ead3c"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:42.865414 12972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:14:42.888509 12987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:42.888737 12972 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 21:14:42.888832 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "285c97d7fe1f40dd8595ebd41a2df6da"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "ef8d1e5946624c2341d34349d4efa0ec"
server_key_iv: "1fae67663a80912671d237ab2f0ead3c"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:42.888940 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:42.907047 12972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:42.909297 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:42.909456 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:42.909745 12972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:42.910099 12972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:42.910135 12972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:42.910164 12972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:42.910180 12972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:42.917761 12972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34987
I20260927 21:14:42.918216 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:14:42.920560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 12972
I20260927 21:14:42.920672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:14:42.920859 392 external_mini_cluster.cc:1428] Setting key c5a734736c4866096bf96963fec58ac6
I20260927 21:14:42.924762 13100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34987 every 8 connection(s)
Sep 27 21:14:42 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:42.936770 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:42.925499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33219 (local address 127.0.98.62:41133)
0927 21:14:42.925548 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:42.925550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:42.926107 (+ 557us) server_negotiation.cc:407] Connection header received
0927 21:14:42.926843 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:42.926844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:42.926858 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:42.926890 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:42.927786 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.928582 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.930193 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.930507 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.930549 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:42.933233 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:42.933235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:42.933235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:42.933244 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:42.935043 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:42.935603 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:42.935604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:42.935605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:42.935670 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:42.935972 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:42.935973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:42.935973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:42.936129 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:42.936215 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:42.936576 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 21:14:42.936666 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:14:42.936983 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:42.925663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41133 (local address 127.0.98.3:33219)
0927 21:14:42.925912 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:42.926002 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:42.926707 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:42.927099 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:42.927104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:42.927381 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:42.927673 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:42.927679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.928708 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.928709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:42.930091 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:42.930095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:42.930658 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:42.930659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:42.930807 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:42.931458 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:42.931469 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:42.933128 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:42.935155 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:42.935165 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:42.935168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:42.935510 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:42.935739 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:42.935740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:42.935741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:42.935899 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:42.936207 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:42.936214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:42.936417 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 21:14:42.936545 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 21:14:42.936724 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":119,"threads_started":1}
I20260927 21:14:42.938026 13101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41133
I20260927 21:14:42.938148 13101 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:42.938356 13101 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260927 21:14:42.938757 12617 ts_manager.cc:194] Registered new tserver with Master: 285c97d7fe1f40dd8595ebd41a2df6da (127.0.98.3:34987)
I20260927 21:14:42.939286 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33219
I20260927 21:14:42.943749 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:14:42.944468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 12698
I20260927 21:14:42.950955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 12835
I20260927 21:14:42.956413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 12972
I20260927 21:14:42.961966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 12585
I20260927 21:14:42.967213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41133
--webserver_interface=127.0.98.62
--webserver_port=45821
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41133
--ranger_config_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44139/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44139/kms with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:43.048336 13104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:43.048564 13104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:43.048586 13104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:43.049849 13104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:14:43.049908 13104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:43.049921 13104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:43.049935 13104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:14:43.049957 13104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:14:43.051689 13104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44139/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41133
--ranger_config_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:43.051941 13104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:43.052165 13104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:43.056890 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:43.056959 13104 server_base.cc:1034] running on GCE node
W20260927 21:14:43.056903 13109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:43.056907 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:43.057329 13104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:43.057616 13104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:43.058771 13104 hybrid_clock.cc:648] HybridClock initialized: now 1790543683058759 us; error 32 us; skew 500 ppm
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:43.061299 13104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:14:43.061789 13104 webserver.cc:460] Webserver started at http://127.0.98.62:45821/ using document root <none> and password file <none>
I20260927 21:14:43.062002 13104 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:43.062058 13104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:43.129618 13104 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:14:43.130460 13120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.130651 13104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:43.130746 13104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "2a2a53fb11874fa599caf70a65be1328"
format_stamp: "Formatted at 2026-09-27 21:14:41 on dist-test-slave-r9x1"
server_key: "a9910c0be42aa34b17cf63969bfa8df4"
server_key_iv: "dbcdc5db5783f97db7d83751ccc7ac4b"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:43.131136 13104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:43.144308 13104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:43.146507 13104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:43.146667 13104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:43.150936 13104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41133
I20260927 21:14:43.150981 13172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41133 every 8 connection(s)
I20260927 21:14:43.151358 13104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:14:43.152673 13173 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:14:43.153249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 13104
I20260927 21:14:43.153441 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Bootstrap starting.
I20260927 21:14:43.153680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39035
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41133
--webserver_port=46739
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44139/kms with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:14:43.162765 13173 log.cc:824] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:43.164667 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:14:43.164968 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Bootstrap complete.
I20260927 21:14:43.167276 13173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260927 21:14:43.167536 13173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2a53fb11874fa599caf70a65be1328, State: Initialized, Role: FOLLOWER
I20260927 21:14:43.167701 13173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260927 21:14:43.167778 13173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:14:43.167829 13173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:14:43.167874 13173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:14:43.168905 13173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260927 21:14:43.169014 13173 leader_election.cc:302] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a2a53fb11874fa599caf70a65be1328; no voters:
I20260927 21:14:43.169193 13173 leader_election.cc:288] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:14:43.169248 13177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:14:43.169417 13177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [term 2 LEADER]: Becoming Leader. State: Replica: 2a2a53fb11874fa599caf70a65be1328, State: Running, Role: LEADER
I20260927 21:14:43.169487 13173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:14:43.169540 13177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260927 21:14:43.169822 13177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a2a53fb11874fa599caf70a65be1328. Latest consensus state: current_term: 2 leader_uuid: "2a2a53fb11874fa599caf70a65be1328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } } }
I20260927 21:14:43.169975 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2a2a53fb11874fa599caf70a65be1328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2a53fb11874fa599caf70a65be1328" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } } }
I20260927 21:14:43.170048 13177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:43.170143 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:43.170574 13186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:14:43.171082 13186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:14:43.171277 13186 catalog_manager.cc:1259] Loaded cluster ID: 28bdcfc201174016b38a2d98871b5f9b
I20260927 21:14:43.171353 13186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:14:43.171423 13173 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:14:43.174945 13186 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:14:43.175633 13186 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Loaded TSK: 0
I20260927 21:14:43.175949 13186 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:14:43.268123 13175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:43.268512 13175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:43.268603 13175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:43.270604 13175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:43.270673 13175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:43.270717 13175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:14:43.273059 13175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46739
--tserver_master_addrs=127.0.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:43.273492 13175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:43.273787 13175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:43.280401 13211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:43.281062 13210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:43.281412 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:43.281682 13175 server_base.cc:1034] running on GCE node
I20260927 21:14:43.282060 13175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:43.282371 13175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:43.284871 13175 hybrid_clock.cc:648] HybridClock initialized: now 1790543683284841 us; error 41 us; skew 500 ppm
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:43.288481 13175 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:14:43.289117 13175 webserver.cc:460] Webserver started at http://127.0.98.1:46739/ using document root <none> and password file <none>
I20260927 21:14:43.289422 13175 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:43.289515 13175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:43.314527 13175 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:14:43.317229 13221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.317544 13175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:14:43.317663 13175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "aaeb6e5a8540439a86c00d0b9125fe97"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "2030eebc6527a6849aa0ecfbe3350944"
server_key_iv: "4234548ed6a65869a9da865c100ab015"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:43.318136 13175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:43.348084 13175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:43.350627 13175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:43.350870 13175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:43.351226 13175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:43.351682 13175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:43.351779 13175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.351848 13175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:43.351893 13175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.370365 13175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39035
I20260927 21:14:43.370462 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39035 every 8 connection(s)
I20260927 21:14:43.371016 13175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:14:43.371758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 13175
I20260927 21:14:43.372066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38863
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41133
--webserver_port=36837
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44139/kms with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:43.396368 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:43.371863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41133 (local address 127.0.98.1:43211)
0927 21:14:43.373749 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:43.373812 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:43.374572 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:43.375298 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:43.375303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:43.375572 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:43.375897 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:43.375905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.377960 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.377961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:43.385546 (+ 7585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:43.385554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.386757 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.386758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:43.386915 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:43.387593 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:43.387605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:43.389115 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:43.394268 (+ 5153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:43.394282 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:43.394285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:43.394653 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:43.394960 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:43.394961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:43.394961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:43.395083 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:43.395587 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:43.395595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:43.395789 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 21:14:43.395882 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:14:43.396057 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1819,"thread_start_us":109,"threads_started":1}
I20260927 21:14:43.397069 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:43.373583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43211 (local address 127.0.98.62:41133)
0927 21:14:43.373848 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:43.373858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:43.373934 (+ 76us) server_negotiation.cc:407] Connection header received
0927 21:14:43.374874 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:43.374883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:43.374964 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:43.375121 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:43.376066 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.377789 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.385726 (+ 7937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.386476 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.386612 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:43.389423 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:43.389430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:43.389431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:43.389445 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:43.394120 (+ 4675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:43.394757 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:43.394761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:43.394761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:43.394894 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:43.395149 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:43.395150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:43.395150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:43.395470 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:43.395542 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:43.396733 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0927 21:14:43.396873 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":96,"threads_started":1}
I20260927 21:14:43.397774 13335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41133
I20260927 21:14:43.397886 13335 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:43.398073 13335 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
W20260927 21:14:43.398947 13335 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41133 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:14:43.399017 13335 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:43.399097 13335 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260927 21:14:43.399422 13335 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:43.399501 13335 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
W20260927 21:14:43.399775 13335 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:14:43.480751 13338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:43.481046 13338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:43.481086 13338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:43.482883 13338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:43.482940 13338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:43.482971 13338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:14:43.484591 13338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36837
--tserver_master_addrs=127.0.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:43.484822 13338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:43.485039 13338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:43.491196 13345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:43.491429 13347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:43.491194 13344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:43.491598 13338 server_base.cc:1034] running on GCE node
I20260927 21:14:43.491801 13338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:43.492094 13338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:43.493302 13338 hybrid_clock.cc:648] HybridClock initialized: now 1790543683493274 us; error 44 us; skew 500 ppm
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:43.496538 13338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:14:43.497107 13338 webserver.cc:460] Webserver started at http://127.0.98.2:36837/ using document root <none> and password file <none>
I20260927 21:14:43.497414 13338 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:43.497498 13338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:43.515793 13338 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260927 21:14:43.516749 13355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.517017 13338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:43.517171 13338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "941c38da39b94b5190cf875526949651"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "81fd4ab174fd32ab8b50501dd653f648"
server_key_iv: "04e74d19367a0c3fbb29b104a3525e27"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:43.517725 13338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:43.534845 13338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:43.536880 13338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:43.537043 13338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:43.537446 13338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:43.537808 13338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:43.537844 13338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.537874 13338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:43.537897 13338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.546303 13338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38863
I20260927 21:14:43.546717 13338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:14:43.547196 13468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38863 every 8 connection(s)
I20260927 21:14:43.551690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 13338
I20260927 21:14:43.551968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34987
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41133
--webserver_port=43607
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44139/kms with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:43.556017 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:43.547505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53871 (local address 127.0.98.62:41133)
0927 21:14:43.547772 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:43.547774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:43.547790 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:14:43.548375 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:43.548376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:43.548393 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:43.548443 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:43.549010 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.549585 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.550588 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.550807 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.550865 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:43.553230 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:43.553232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:43.553232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:43.553241 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:43.554656 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:43.555075 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:43.555076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:43.555077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:43.555131 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:43.555365 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:43.555366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:43.555366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:43.555493 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:43.555547 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:43.555854 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 21:14:43.555945 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260927 21:14:43.556527 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:43.547510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41133 (local address 127.0.98.2:53871)
0927 21:14:43.547676 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:43.547722 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:43.548262 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:43.548514 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:43.548517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:43.548711 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:43.548912 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:43.548916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.549674 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.549675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:43.550501 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:43.550504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.550899 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.550900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:43.551017 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:43.551578 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:43.551589 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:43.553104 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:43.554741 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:43.554749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:43.554753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:43.555003 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:43.555187 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:43.555188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:43.555188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:43.555310 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:43.555549 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:43.555556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:43.555714 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 21:14:43.556170 (+ 456us) client_negotiation.cc:240] Negotiation successful
0927 21:14:43.556352 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260927 21:14:43.556977 13469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41133
I20260927 21:14:43.557096 13469 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:43.557335 13469 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
W20260927 21:14:43.557991 13469 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41133 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:14:43.558056 13469 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:43.558138 13469 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260927 21:14:43.558431 13469 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:43.558506 13469 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
W20260927 21:14:43.558848 13469 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:14:43.680554 13472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:43.680926 13472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:43.680967 13472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:43.682976 13472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:43.683050 13472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:43.683094 13472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:14:43.685757 13472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43607
--tserver_master_addrs=127.0.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:43.686056 13472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:43.686357 13472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:43.691983 13479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:43.691983 13478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:43.692124 13481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:43.692550 13472 server_base.cc:1034] running on GCE node
I20260927 21:14:43.692807 13472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:43.693135 13472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:43.694346 13472 hybrid_clock.cc:648] HybridClock initialized: now 1790543683694319 us; error 44 us; skew 500 ppm
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:43.697724 13472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:14:43.698369 13472 webserver.cc:460] Webserver started at http://127.0.98.3:43607/ using document root <none> and password file <none>
I20260927 21:14:43.698607 13472 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:43.698675 13472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:43.715619 13472 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:43.716509 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.716670 13472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:43.716755 13472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "285c97d7fe1f40dd8595ebd41a2df6da"
format_stamp: "Formatted at 2026-09-27 21:14:42 on dist-test-slave-r9x1"
server_key: "ef8d1e5946624c2341d34349d4efa0ec"
server_key_iv: "1fae67663a80912671d237ab2f0ead3c"
server_key_version: "kuduclusterkey@0"
I20260927 21:14:43.717224 13472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:43.732420 13472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:43.734478 13472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:43.734671 13472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:43.734910 13472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:43.735313 13472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:43.735356 13472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.735378 13472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:43.735390 13472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:43.741688 13472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34987
I20260927 21:14:43.741860 13602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34987 every 8 connection(s)
I20260927 21:14:43.742105 13472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:14:43 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:43.751129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 13472
I20260927 21:14:43.753713 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:43.743223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35529 (local address 127.0.98.62:41133)
0927 21:14:43.743267 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:43.743269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:43.743280 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:14:43.743779 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:43.743780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:43.743794 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:43.743844 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:43.744425 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.744961 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.746532 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.746851 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.746920 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:43.749814 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:43.749817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:43.749817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:43.749826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:43.751574 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:43.752607 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:43.752609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:43.752609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:43.752673 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:43.752971 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:43.752973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:43.752973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:43.753115 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:43.753154 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:43.753524 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 21:14:43.753613 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:14:43.753898 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:43.742970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41133 (local address 127.0.98.3:35529)
0927 21:14:43.743124 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:43.743168 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:43.743675 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:43.743917 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:43.743921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:43.744134 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:43.744333 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:43.744338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.745075 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.745077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:43.746427 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:43.746431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.746968 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.746969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:43.747120 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:43.747673 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:43.747685 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:43.749675 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:43.752156 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:43.752167 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:43.752170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:43.752500 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:43.752748 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:43.752749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:43.752750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:43.752895 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:43.753190 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:43.753195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:43.753390 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 21:14:43.753489 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 21:14:43.753661 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260927 21:14:43.754503 13603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41133
I20260927 21:14:43.754606 13603 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:43.754786 13603 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
W20260927 21:14:43.755400 13603 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41133 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:14:43.755525 13603 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:43.755642 13603 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260927 21:14:43.756001 13603 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:43.756079 13603 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
W20260927 21:14:43.756366 13603 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:14:43.765300 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:43.752045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.0.98.62:41133)
0927 21:14:43.752185 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:43.752187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:43.752197 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:14:43.752219 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:43.752220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:43.752233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:43.752315 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:43.755309 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.756037 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.756943 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:43.757287 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:43.757359 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:43.758828 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:43.758830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:43.758830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:43.758837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:43.763980 (+ 5143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:43.764436 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:43.764437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:43.764437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:43.764493 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:43.764711 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:43.764712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:43.764712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:43.764841 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:43.765008 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:43.765070 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 21:14:43.765160 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
Sep 27 21:14:44 dist-test-slave-r9x1 krb5kdc[12141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:44.399983 13335 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:44.400168 13335 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260927 21:14:44.400585 13137 ts_manager.cc:194] Registered new tserver with Master: aaeb6e5a8540439a86c00d0b9125fe97 (127.0.98.1:39035)
I20260927 21:14:44.401327 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43211
I20260927 21:14:44.559089 13469 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:44.559271 13469 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260927 21:14:44.559895 13137 ts_manager.cc:194] Registered new tserver with Master: 941c38da39b94b5190cf875526949651 (127.0.98.2:38863)
I20260927 21:14:44.560444 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53871
Sep 27 21:14:44 dist-test-slave-r9x1 krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:14:44.756556 13603 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:44.756726 13603 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260927 21:14:44.757004 13136 ts_manager.cc:194] Registered new tserver with Master: 285c97d7fe1f40dd8595ebd41a2df6da (127.0.98.3:34987)
I20260927 21:14:44.757602 13136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35529
I20260927 21:14:44.759274 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:14:44.771049 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.760328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.0.98.62:41133)
0927 21:14:44.760936 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:44.760938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:44.760948 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:14:44.760977 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:44.760978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:44.760991 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:44.761059 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:44.762933 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.763667 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.764437 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.764655 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.764714 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.766908 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:44.766911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:44.766911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:44.766921 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:44.769212 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:44.769698 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:44.769699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:44.769700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:44.769796 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:44.769999 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:44.770000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:44.770001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:44.770194 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:44.770286 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:44.770845 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 21:14:44.770938 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260927 21:14:44.773594 13136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:14:44.776017 13136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:14:44.793013 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.785708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39035 (local address 127.0.0.1:40632)
0927 21:14:44.788787 (+ 3079us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:44.788801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:44.788907 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:44.789216 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:44.789220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:44.789222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:44.789340 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.789343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.790385 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.790386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.791278 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.791280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.792307 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.792308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.792756 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.792788 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:14:44.792862 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:14:44.792926 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3057,"thread_start_us":59,"threads_started":1}
I20260927 21:14:44.793815 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.785901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.0.98.1:39035)
0927 21:14:44.786317 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:44.786323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:44.788874 (+ 2551us) server_negotiation.cc:407] Connection header received
0927 21:14:44.788989 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:44.788992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:44.789009 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:44.789062 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:44.789433 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.790305 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.791365 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.792102 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.792194 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.792266 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:44.793676 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0927 21:14:44.793732 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":53,"threads_started":1}
I20260927 21:14:44.794595 13262 tablet_service.cc:1467] Processing CreateTablet for tablet 596cd4c76c9640ed994611493433640a (DEFAULT_TABLE table=test-table [id=18b7053c98474740b24aed555b4b5313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:14:44.794983 13262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596cd4c76c9640ed994611493433640a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:44.797968 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.786580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34987 (local address 127.0.0.1:46434)
0927 21:14:44.793207 (+ 6627us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:44.793218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:44.793289 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:44.793609 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:44.793610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:44.793612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:44.793660 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.793662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.795054 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.795055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.796569 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.796574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.797701 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.797702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.797771 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.797776 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:14:44.797831 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:14:44.797883 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6614,"thread_start_us":78,"threads_started":1}
I20260927 21:14:44.798316 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.786836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.0.98.3:34987)
0927 21:14:44.787275 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:44.787280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:44.793355 (+ 6075us) server_negotiation.cc:407] Connection header received
0927 21:14:44.793400 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:44.793403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:44.793420 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:44.793512 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:44.793738 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.794934 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.796674 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.797509 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.797582 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.797661 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:44.798180 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 21:14:44.798236 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":44,"threads_started":1}
I20260927 21:14:44.799003 13537 tablet_service.cc:1467] Processing CreateTablet for tablet 596cd4c76c9640ed994611493433640a (DEFAULT_TABLE table=test-table [id=18b7053c98474740b24aed555b4b5313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:14:44.799616 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596cd4c76c9640ed994611493433640a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:44.803098 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.785473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38863 (local address 127.0.0.1:36744)
0927 21:14:44.786064 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:44.786115 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:44.786187 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:44.793585 (+ 7398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:44.793586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:44.793589 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:44.793638 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.793640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.800463 (+ 6823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.800465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.801724 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.801728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.802815 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.802816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.802893 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.802897 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:14:44.802944 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:14:44.802995 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":74,"threads_started":1}
I20260927 21:14:44.803619 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.785569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.0.98.2:38863)
0927 21:14:44.793374 (+ 7805us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:44.793379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:44.793395 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:14:44.793435 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:44.793438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:44.793454 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:44.793561 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:44.793725 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.800338 (+ 6613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.801838 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.802620 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.802694 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.802771 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:44.803478 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 21:14:44.803535 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7747,"thread_start_us":70,"threads_started":1}
I20260927 21:14:44.804365 13637 tablet_bootstrap.cc:492] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: Bootstrap starting.
I20260927 21:14:44.804451 13403 tablet_service.cc:1467] Processing CreateTablet for tablet 596cd4c76c9640ed994611493433640a (DEFAULT_TABLE table=test-table [id=18b7053c98474740b24aed555b4b5313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:14:44.804790 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596cd4c76c9640ed994611493433640a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:44.805160 13637 tablet_bootstrap.cc:654] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:44.805526 13637 log.cc:824] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:44.808908 13639 tablet_bootstrap.cc:492] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Bootstrap starting.
I20260927 21:14:44.809428 13636 tablet_bootstrap.cc:492] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97: Bootstrap starting.
I20260927 21:14:44.809705 13639 tablet_bootstrap.cc:654] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:44.810102 13639 log.cc:824] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:44.810176 13636 tablet_bootstrap.cc:654] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:44.810524 13636 log.cc:824] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:44.810961 13639 tablet_bootstrap.cc:492] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: No bootstrap required, opened a new log
I20260927 21:14:44.811045 13639 ts_tablet_manager.cc:1397] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:14:44.811265 13636 tablet_bootstrap.cc:492] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97: No bootstrap required, opened a new log
I20260927 21:14:44.811321 13636 ts_tablet_manager.cc:1397] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:14:44.813884 13636 raft_consensus.cc:348] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } }
I20260927 21:14:44.813845 13639 raft_consensus.cc:348] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } }
I20260927 21:14:44.814059 13639 raft_consensus.cc:374] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:44.814057 13636 raft_consensus.cc:374] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:44.814096 13639 raft_consensus.cc:729] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941c38da39b94b5190cf875526949651, State: Initialized, Role: FOLLOWER
I20260927 21:14:44.814101 13636 raft_consensus.cc:729] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaeb6e5a8540439a86c00d0b9125fe97, State: Initialized, Role: FOLLOWER
I20260927 21:14:44.814256 13636 consensus_queue.cc:260] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } }
I20260927 21:14:44.814548 13636 ts_tablet_manager.cc:1428] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 21:14:44.814698 13335 heartbeater.cc:503] Master 127.0.98.62:41133 was elected leader, sending a full tablet report...
I20260927 21:14:44.815320 13639 consensus_queue.cc:260] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } }
I20260927 21:14:44.815733 13639 ts_tablet_manager.cc:1428] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260927 21:14:44.815898 13469 heartbeater.cc:503] Master 127.0.98.62:41133 was elected leader, sending a full tablet report...
I20260927 21:14:44.818126 13637 tablet_bootstrap.cc:492] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: No bootstrap required, opened a new log
I20260927 21:14:44.818225 13637 ts_tablet_manager.cc:1397] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.001s
I20260927 21:14:44.821902 13637 raft_consensus.cc:348] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } }
I20260927 21:14:44.822149 13637 raft_consensus.cc:374] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:44.822216 13637 raft_consensus.cc:729] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285c97d7fe1f40dd8595ebd41a2df6da, State: Initialized, Role: FOLLOWER
I20260927 21:14:44.822402 13637 consensus_queue.cc:260] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } }
I20260927 21:14:44.822747 13637 ts_tablet_manager.cc:1428] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 21:14:44.822904 13603 heartbeater.cc:503] Master 127.0.98.62:41133 was elected leader, sending a full tablet report...
W20260927 21:14:44.877034 13336 tablet.cc:2369] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:14:44.949039 13642 raft_consensus.cc:482] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:14:44.949210 13642 raft_consensus.cc:504] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } }
I20260927 21:14:44.949661 13642 leader_election.cc:288] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 941c38da39b94b5190cf875526949651 (127.0.98.2:38863), 285c97d7fe1f40dd8595ebd41a2df6da (127.0.98.3:34987)
I20260927 21:14:44.954082 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.950125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38937 (local address 127.0.98.2:38863)
0927 21:14:44.950170 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:44.950172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:44.950360 (+ 188us) server_negotiation.cc:407] Connection header received
0927 21:14:44.950463 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:44.950464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:44.950476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:44.950504 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:44.950746 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.951513 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.953071 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.953677 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.953725 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.953738 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:44.953942 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 21:14:44.953987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:14:44.954125 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.950142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55505 (local address 127.0.98.3:34987)
0927 21:14:44.950198 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:44.950200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:44.950364 (+ 164us) server_negotiation.cc:407] Connection header received
0927 21:14:44.950468 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:44.950469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:44.950483 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:44.950513 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:44.950770 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.951496 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.953061 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.953690 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.953741 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.953752 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:44.953988 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 21:14:44.954035 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:14:44.954394 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.949914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34987 (local address 127.0.98.1:55505)
0927 21:14:44.950295 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:44.950306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:44.950393 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:44.950582 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:44.950583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:44.950585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:44.950680 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.950682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.951611 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.951612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.952948 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.952952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.953830 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.953831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.953907 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.953911 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:14:44.954256 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 21:14:44.954310 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":46,"thread_start_us":103,"threads_started":1}
I20260927 21:14:44.954721 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596cd4c76c9640ed994611493433640a" candidate_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" is_pre_election: true
I20260927 21:14:44.954918 13557 raft_consensus.cc:2454] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaeb6e5a8540439a86c00d0b9125fe97 in term 0.
I20260927 21:14:44.955168 13225 leader_election.cc:302] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285c97d7fe1f40dd8595ebd41a2df6da, aaeb6e5a8540439a86c00d0b9125fe97; no voters:
I20260927 21:14:44.955310 13642 raft_consensus.cc:2781] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:14:44.955343 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.949885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38863 (local address 127.0.98.1:38937)
0927 21:14:44.950288 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:44.950300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:44.950388 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:44.950572 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:44.950573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:44.950575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:44.950672 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.950675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.951616 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.951617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.952943 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:44.952946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.953799 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.953799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:44.953866 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.953870 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:14:44.954563 (+ 693us) client_negotiation.cc:240] Negotiation successful
0927 21:14:44.954611 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":48,"thread_start_us":113,"threads_started":1}
I20260927 21:14:44.955363 13642 raft_consensus.cc:482] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:14:44.955591 13642 raft_consensus.cc:3037] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:44.955876 13423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596cd4c76c9640ed994611493433640a" candidate_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941c38da39b94b5190cf875526949651" is_pre_election: true
I20260927 21:14:44.956033 13423 raft_consensus.cc:2454] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaeb6e5a8540439a86c00d0b9125fe97 in term 0.
I20260927 21:14:44.956533 13642 raft_consensus.cc:504] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } }
I20260927 21:14:44.956687 13642 leader_election.cc:288] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [CANDIDATE]: Term 1 election: Requested vote from peers 941c38da39b94b5190cf875526949651 (127.0.98.2:38863), 285c97d7fe1f40dd8595ebd41a2df6da (127.0.98.3:34987)
I20260927 21:14:44.956868 13423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596cd4c76c9640ed994611493433640a" candidate_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941c38da39b94b5190cf875526949651"
I20260927 21:14:44.956889 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596cd4c76c9640ed994611493433640a" candidate_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285c97d7fe1f40dd8595ebd41a2df6da"
I20260927 21:14:44.956939 13423 raft_consensus.cc:3037] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:44.956966 13557 raft_consensus.cc:3037] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:44.957831 13423 raft_consensus.cc:2454] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaeb6e5a8540439a86c00d0b9125fe97 in term 1.
I20260927 21:14:44.957870 13557 raft_consensus.cc:2454] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaeb6e5a8540439a86c00d0b9125fe97 in term 1.
I20260927 21:14:44.958007 13223 leader_election.cc:302] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941c38da39b94b5190cf875526949651, aaeb6e5a8540439a86c00d0b9125fe97; no voters:
I20260927 21:14:44.958117 13642 raft_consensus.cc:2781] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:14:44.958290 13642 raft_consensus.cc:686] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [term 1 LEADER]: Becoming Leader. State: Replica: aaeb6e5a8540439a86c00d0b9125fe97, State: Running, Role: LEADER
I20260927 21:14:44.958396 13642 consensus_queue.cc:237] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } }
I20260927 21:14:44.959272 13137 catalog_manager.cc:5495] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaeb6e5a8540439a86c00d0b9125fe97 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaeb6e5a8540439a86c00d0b9125fe97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 } health_report { overall_health: UNKNOWN } } }
I20260927 21:14:44.974438 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:44.971116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.0.98.1:39035)
0927 21:14:44.971163 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:44.971165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:44.971175 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:14:44.971279 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:44.971279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:44.971304 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:44.971378 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:14:44.971675 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.972370 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.973484 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:44.973773 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:44.973830 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:44.974033 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:14:44.974201 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:14:44.974304 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:44.974323 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 21:14:44.974352 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:14:44.976761 13423 raft_consensus.cc:1261] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 1 FOLLOWER]: Refusing update from remote peer aaeb6e5a8540439a86c00d0b9125fe97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:14:44.976761 13557 raft_consensus.cc:1261] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [term 1 FOLLOWER]: Refusing update from remote peer aaeb6e5a8540439a86c00d0b9125fe97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:14:44.977252 13647 consensus_queue.cc:1035] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "941c38da39b94b5190cf875526949651" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:14:44.977427 13642 consensus_queue.cc:1035] T 596cd4c76c9640ed994611493433640a P aaeb6e5a8540439a86c00d0b9125fe97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285c97d7fe1f40dd8595ebd41a2df6da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:14:44.982230 13651 mvcc.cc:205] Tried to move back new op lower bound from 7334066933661487104 to 7334066933590437888. Current Snapshot: MvccSnapshot[applied={T|T < 7334066933661487104}]
I20260927 21:14:44.983260 13650 mvcc.cc:205] Tried to move back new op lower bound from 7334066933661487104 to 7334066933590437888. Current Snapshot: MvccSnapshot[applied={T|T < 7334066933661487104 or (T in {7334066933661487104})}]
I20260927 21:14:44.984750 13137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:14:44.984862 13137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:14:44.985613 13137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a2a53fb11874fa599caf70a65be1328: Sending DeleteTablet for 3 replicas of tablet 596cd4c76c9640ed994611493433640a
I20260927 21:14:44.986007 13537 tablet_service.cc:1514] Processing DeleteTablet for tablet 596cd4c76c9640ed994611493433640a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46434
I20260927 21:14:44.986096 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet 596cd4c76c9640ed994611493433640a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36744
I20260927 21:14:44.986328 13262 tablet_service.cc:1514] Processing DeleteTablet for tablet 596cd4c76c9640ed994611493433640a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40632
I20260927 21:14:44.986951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 13175
I20260927 21:14:44.988269 13660 tablet_replica.cc:330] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: stopping tablet replica
I20260927 21:14:44.988377 13660 raft_consensus.cc:2229] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:14:44.988515 13660 pending_rounds.cc:62] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Trying to abort 2 pending ops.
I20260927 21:14:44.988546 13660 pending_rounds.cc:68] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334066933661487104 op_type: WRITE_OP write_request { tablet_id: "596cd4c76c9640ed994611493433640a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\360\220\346\325\006\"8\n\ntest-admin\022*\n 18b7053c98474740b24aed555b4b5313\020\001\030\001 \001(\001" signature: "<redacted>""\2128\301N\204@P\343F\362\027\374\214\204\027;\347\377$#\254\3679\346\277~\350O\233\215\226_|e\310W<\203\325\035V\375\333\367\315\0142n;\014D\255\305N\261\325@]\255\365\312\326\361\014" signing_key_seq_num: 0 } } request_id { client_id: "2e393988faeb47dd85bd4d1721219a52" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 21:14:44.988754 13660 pending_rounds.cc:68] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334066933590437888 op_type: NO_OP noop_request { }
I20260927 21:14:44.988790 13660 raft_consensus.cc:2866] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 21:14:44.988813 13660 raft_consensus.cc:2258] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:14:44.989257 13660 ts_tablet_manager.cc:1905] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:14:44.993214 13660 ts_tablet_manager.cc:1918] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:14:44.993346 13660 log.cc:1196] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/596cd4c76c9640ed994611493433640a
I20260927 21:14:44.994052 13660 ts_tablet_manager.cc:1939] T 596cd4c76c9640ed994611493433640a P 941c38da39b94b5190cf875526949651: Deleting consensus metadata
W20260927 21:14:44.994459 13358 connection.cc:535] server connection from 127.0.98.1:38937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:14:44.994684 13122 catalog_manager.cc:4841] TS 941c38da39b94b5190cf875526949651 (127.0.98.2:38863): tablet 596cd4c76c9640ed994611493433640a (table test-table [id=18b7053c98474740b24aed555b4b5313]) successfully deleted
I20260927 21:14:44.994879 13659 tablet_replica.cc:330] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: stopping tablet replica
W20260927 21:14:44.994979 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39035 (error 108)
I20260927 21:14:44.995004 13659 raft_consensus.cc:2229] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:14:44.995136 13659 raft_consensus.cc:2258] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:14:44.995651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 13338
I20260927 21:14:44.995675 13659 ts_tablet_manager.cc:1905] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:14:44.997385 13659 ts_tablet_manager.cc:1918] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:14:44.997458 13659 log.cc:1196] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/596cd4c76c9640ed994611493433640a
I20260927 21:14:44.997715 13659 ts_tablet_manager.cc:1939] T 596cd4c76c9640ed994611493433640a P 285c97d7fe1f40dd8595ebd41a2df6da: Deleting consensus metadata
I20260927 21:14:45.000542 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:45.000340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39035 (local address 127.0.0.1:40654)
0927 21:14:45.000447 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:45.000497 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 21:14:45.000718 13122 catalog_manager.cc:4584] TS aaeb6e5a8540439a86c00d0b9125fe97 (127.0.98.1:39035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 596cd4c76c9640ed994611493433640a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39035: connect: Connection refused (error 111)
I20260927 21:14:45.000857 13124 catalog_manager.cc:4841] TS 285c97d7fe1f40dd8595ebd41a2df6da (127.0.98.3:34987): tablet 596cd4c76c9640ed994611493433640a (table test-table [id=18b7053c98474740b24aed555b4b5313]) successfully deleted
I20260927 21:14:45.001161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 13472
I20260927 21:14:45.007769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 13104
I20260927 21:14:45.013315 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:14:45.348155 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:14:45.348263 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:14:45.683168 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:14:45.683 UTC [12171] LOG: received smart shutdown request
2026-09-27 21:14:45.687 UTC [12171] LOG: background worker "logical replication launcher" (PID 12180) exited with exit code 1
2026-09-27 21:14:45.687 UTC [12175] LOG: shutting down
2026-09-27 21:14:45.873 UTC [12171] LOG: database system is shut down
2026-09-27T21:14:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50245 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:14:46 dist-test-slave-r9x1 krb5kdc[13672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:14:46 dist-test-slave-r9x1 krb5kdc[13672](info): set up 2 sockets
Sep 27 21:14:46 dist-test-slave-r9x1 krb5kdc[13672](info): commencing operation
krb5kdc: starting...
W20260927 21:14:46.924937 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:14:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:14:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:14:47.045552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39683
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39683,127.0.98.61:37355,127.0.98.60:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:47.126719 13685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:47.126952 13685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:47.126984 13685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:47.128202 13685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:14:47.128249 13685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:47.128268 13685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:47.128283 13685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:14:47.128298 13685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:14:47.129956 13685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39683,127.0.98.61:37355,127.0.98.60:37347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:47.130230 13685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:47.130447 13685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:47.135227 13690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:47.135227 13691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:47.135244 13693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:47.135388 13685 server_base.cc:1034] running on GCE node
I20260927 21:14:47.135771 13685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:47.136061 13685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:47.137213 13685 hybrid_clock.cc:648] HybridClock initialized: now 1790543687137200 us; error 30 us; skew 500 ppm
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:47.139663 13685 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:14:47.140169 13685 webserver.cc:460] Webserver started at http://127.0.98.62:34785/ using document root <none> and password file <none>
I20260927 21:14:47.140381 13685 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:47.140430 13685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:47.140570 13685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:14:47.141726 13685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "3e3202f2bbce4520b8d18d376b2b3311"
format_stamp: "Formatted at 2026-09-27 21:14:47 on dist-test-slave-r9x1"
server_key: "4528f2bcba3afd8d2019e744633299a9"
server_key_iv: "3a1d4f523d901adf9442d048b7d6db73"
server_key_version: "encryptionkey@0"
I20260927 21:14:47.142081 13685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e3202f2bbce4520b8d18d376b2b3311"
format_stamp: "Formatted at 2026-09-27 21:14:47 on dist-test-slave-r9x1"
server_key: "4528f2bcba3afd8d2019e744633299a9"
server_key_iv: "3a1d4f523d901adf9442d048b7d6db73"
server_key_version: "encryptionkey@0"
I20260927 21:14:47.143409 13685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:14:47.144203 13700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:47.144380 13685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:47.144465 13685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "3e3202f2bbce4520b8d18d376b2b3311"
format_stamp: "Formatted at 2026-09-27 21:14:47 on dist-test-slave-r9x1"
server_key: "4528f2bcba3afd8d2019e744633299a9"
server_key_iv: "3a1d4f523d901adf9442d048b7d6db73"
server_key_version: "encryptionkey@0"
I20260927 21:14:47.144537 13685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:47.154218 13685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:47.156108 13685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:47.156252 13685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:47.160449 13685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39683
I20260927 21:14:47.160561 13752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39683 every 8 connection(s)
I20260927 21:14:47.160861 13685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:14:47.161773 13753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:47.163412 13753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:47.170641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 13685
I20260927 21:14:47.170755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:14:47.170979 392 external_mini_cluster.cc:1428] Setting key 6f02d8969010d7a70a33cd6e4918b383
I20260927 21:14:47.173444 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.164400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.0.98.62:39683)
0927 21:14:47.164538 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.164542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.164574 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:14:47.165081 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.165084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.165104 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.165207 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.165867 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.166723 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.167658 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.168001 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.168097 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.170273 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.170281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.170281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.170289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.171893 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.172254 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.172258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.172258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.172339 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.172531 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.172532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.172532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.172755 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.172827 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.173148 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.173234 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260927 21:14:47.173450 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.164201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39683 (local address 127.0.0.1:47944)
0927 21:14:47.164392 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.164442 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:47.164926 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:47.165258 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:47.165261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:47.165579 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:47.165753 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.165757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.166821 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.166822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.167569 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.167572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.168114 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.168115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.168244 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.168716 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:47.168724 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:47.170188 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:47.171982 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.171989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.171992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.172182 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.172397 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.172398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.172398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.172479 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.172818 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:47.172822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:47.172889 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 21:14:47.172976 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:14:47.173142 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260927 21:14:47.174286 13753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:14:47.174703 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.174562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37355 (local address 127.0.0.1:52510)
0927 21:14:47.174611 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.174670 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:14:47.174831 13703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
I20260927 21:14:47.175246 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.175161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37355 (local address 127.0.0.1:52522)
0927 21:14:47.175198 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.175224 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 21:14:47.175352 13753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37355: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 21:14:47.197324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37355
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39683,127.0.98.61:37355,127.0.98.60:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:14:47.224627 13753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } attempt: 1
I20260927 21:14:47.225164 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.225034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37355 (local address 127.0.0.1:52528)
0927 21:14:47.225095 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.225129 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 21:14:47.225648 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.225509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37355 (local address 127.0.0.1:52542)
0927 21:14:47.225581 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.225618 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:14:47.225749 13753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37355: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
W20260927 21:14:47.280105 13760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:47.280447 13760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:47.280478 13760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:47.281873 13760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:14:47.281924 13760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:47.281937 13760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:47.281948 13760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:14:47.281960 13760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:14:47.283681 13760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39683,127.0.98.61:37355,127.0.98.60:37347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:47.283972 13760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:47.284196 13760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:47.289572 13768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:47.289608 13760 server_base.cc:1034] running on GCE node
W20260927 21:14:47.289577 13765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:47.289620 13766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:47.290225 13760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:47.290565 13760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:47.291723 13760 hybrid_clock.cc:648] HybridClock initialized: now 1790543687291708 us; error 38 us; skew 500 ppm
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:47.293886 13753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } attempt: 2
I20260927 21:14:47.294286 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.294155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37355 (local address 127.0.0.1:52552)
0927 21:14:47.294218 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.294253 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:14:47.294395 13760 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 21:14:47.294607 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.294521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37355 (local address 127.0.0.1:52564)
0927 21:14:47.294555 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.294585 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 21:14:47.294695 13753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37355: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37355: connect: Connection refused (error 111)
I20260927 21:14:47.294943 13760 webserver.cc:460] Webserver started at http://127.0.98.61:33003/ using document root <none> and password file <none>
I20260927 21:14:47.295150 13760 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:47.295189 13760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:47.295284 13760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:14:47.296335 13760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/data/instance:
uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500"
format_stamp: "Formatted at 2026-09-27 21:14:47 on dist-test-slave-r9x1"
server_key: "aa073979c3869ae31043e71b2150cba3"
server_key_iv: "a63847ab01290fb7f545c35991086572"
server_key_version: "encryptionkey@0"
I20260927 21:14:47.296653 13760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/wal/instance:
uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500"
format_stamp: "Formatted at 2026-09-27 21:14:47 on dist-test-slave-r9x1"
server_key: "aa073979c3869ae31043e71b2150cba3"
server_key_iv: "a63847ab01290fb7f545c35991086572"
server_key_version: "encryptionkey@0"
I20260927 21:14:47.297996 13760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:14:47.298729 13775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:47.298877 13760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:14:47.298960 13760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/wal
uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500"
format_stamp: "Formatted at 2026-09-27 21:14:47 on dist-test-slave-r9x1"
server_key: "aa073979c3869ae31043e71b2150cba3"
server_key_iv: "a63847ab01290fb7f545c35991086572"
server_key_version: "encryptionkey@0"
I20260927 21:14:47.299039 13760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:47.306394 13760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:47.308408 13760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:47.308537 13760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:47.312299 13760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37355
I20260927 21:14:47.312330 13827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37355 every 8 connection(s)
I20260927 21:14:47.312664 13760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:14:47.313618 13828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:47.315236 13828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:47.322351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 13760
I20260927 21:14:47.322472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-1/wal/instance
I20260927 21:14:47.322651 392 external_mini_cluster.cc:1428] Setting key 802d1353e9acb0c93a69cd310b7ae189
I20260927 21:14:47.324923 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.315953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.0.98.62:39683)
0927 21:14:47.316032 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.316034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.316201 (+ 167us) server_negotiation.cc:407] Connection header received
0927 21:14:47.316670 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.316671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.316682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.316727 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.317339 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.317828 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.318853 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.319067 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.319132 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.321919 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.321921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.321922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.321931 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.323409 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.323986 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.323987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.323988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.324068 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.324285 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.324286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.324286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.324430 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.324498 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.324744 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.324829 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:14:47.325026 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.315920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39683 (local address 127.0.0.1:47946)
0927 21:14:47.316086 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.316124 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:47.316595 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:47.316787 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:47.316791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:47.317049 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:47.317246 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.317251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.317903 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.317903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.318750 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.318753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.319253 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.319255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.319459 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.320227 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:47.320238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:47.321798 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:47.323507 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.323526 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.323528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.323892 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.324132 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.324133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.324133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.324234 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.324485 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:47.324488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:47.324627 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 21:14:47.324708 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 21:14:47.324843 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260927 21:14:47.325649 13828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:14:47.335350 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.326012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.0.98.61:37355)
0927 21:14:47.326186 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.326190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.326206 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:14:47.326222 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.326223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.326234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.326293 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.326568 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.327256 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.327906 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.328209 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.328268 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.332745 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.332756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.332757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.332765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.334278 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.334652 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.334655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.334656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.334706 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.334870 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.334870 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.334870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.335006 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.335061 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.335192 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.335262 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
I20260927 21:14:47.335435 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.325938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37355 (local address 127.0.0.1:52568)
0927 21:14:47.325972 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.325982 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:47.326018 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:47.326305 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:47.326306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:47.326438 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:47.326491 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.326492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.327329 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.327330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.327837 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.327839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.328274 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.328275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.328335 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.328788 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:47.328792 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:47.332625 (+ 3833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:47.334380 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.334382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.334382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.334580 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.334761 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.334762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.334762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.334821 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.335056 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:47.335057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:47.335104 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:14:47.335169 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 21:14:47.335319 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":51}
I20260927 21:14:47.336285 13828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:14:47.336637 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.336497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43782)
0927 21:14:47.336547 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.336606 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:14:47.336772 13778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
I20260927 21:14:47.337189 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.337097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43796)
0927 21:14:47.337141 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.337169 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:14:47.337303 13828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 21:14:47.347658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37347
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39683,127.0.98.61:37355,127.0.98.60:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:14:47.385903 13753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } attempt: 3
I20260927 21:14:47.386533 13828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } attempt: 1
I20260927 21:14:47.386958 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.386817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43808)
0927 21:14:47.386878 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.386924 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 21:14:47.387315 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.387200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43824)
0927 21:14:47.387246 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.387281 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:14:47.387420 13828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:14:47.393251 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.386387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.0.98.61:37355)
0927 21:14:47.386457 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.386459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.386469 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:14:47.386498 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.386500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.386519 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.386580 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.386958 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.387642 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.388345 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.388568 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.388625 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.390879 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.390880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.390880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.390888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.392183 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.392497 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.392498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.392498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.392551 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.392713 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.392714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.392714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.392846 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.392904 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.393070 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.393152 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:14:47.393289 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.386307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37355 (local address 127.0.0.1:52582)
0927 21:14:47.386386 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.386398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:47.386449 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:47.386591 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:47.386592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:47.386785 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:47.386886 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.386890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.387727 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.387728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.388265 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.388267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.388633 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.388634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.388711 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.389251 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:47.389256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:47.390770 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:47.392262 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.392264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.392264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.392432 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.392605 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.392606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.392607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.392665 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.392904 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:47.392905 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:47.392982 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 21:14:47.393037 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:14:47.393151 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 21:14:47.393976 13753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:14:47.394320 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.394223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43836)
0927 21:14:47.394263 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.394291 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 21:14:47.394619 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.394561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43850)
0927 21:14:47.394585 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.394605 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 21:14:47.394694 13753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
I20260927 21:14:47.425854 13753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } attempt: 1
I20260927 21:14:47.426415 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.426254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43852)
0927 21:14:47.426318 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.426378 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:14:47.426836 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.426741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43864)
0927 21:14:47.426783 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.426812 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 21:14:47.426944 13753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
W20260927 21:14:47.431747 13835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:47.431972 13835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:47.431998 13835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:47.433228 13835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:14:47.433302 13835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:47.433324 13835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:47.433339 13835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:14:47.433350 13835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:14:47.434957 13835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39683,127.0.98.61:37355,127.0.98.60:37347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:47.435190 13835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:47.435412 13835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:47.440176 13840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:47.440188 13843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:47.440234 13835 server_base.cc:1034] running on GCE node
W20260927 21:14:47.440189 13841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:47.440623 13835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:47.440882 13835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:47.442027 13835 hybrid_clock.cc:648] HybridClock initialized: now 1790543687442006 us; error 36 us; skew 500 ppm
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:47.444403 13835 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 21:14:47.444847 13835 webserver.cc:460] Webserver started at http://127.0.98.60:44991/ using document root <none> and password file <none>
I20260927 21:14:47.445050 13835 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:47.445101 13835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:47.445225 13835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:14:47.446285 13835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/data/instance:
uuid: "ec676f0a35774a17a1cc0d0b617ee109"
format_stamp: "Formatted at 2026-09-27 21:14:47 on dist-test-slave-r9x1"
server_key: "736494fba9364f5987a9e998ea9464cb"
server_key_iv: "9138116f0306bcddb28e4bf553c35ad0"
server_key_version: "encryptionkey@0"
I20260927 21:14:47.446611 13835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/wal/instance:
uuid: "ec676f0a35774a17a1cc0d0b617ee109"
format_stamp: "Formatted at 2026-09-27 21:14:47 on dist-test-slave-r9x1"
server_key: "736494fba9364f5987a9e998ea9464cb"
server_key_iv: "9138116f0306bcddb28e4bf553c35ad0"
server_key_version: "encryptionkey@0"
I20260927 21:14:47.447919 13835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:14:47.448558 13850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:47.448693 13835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:14:47.448771 13835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/wal
uuid: "ec676f0a35774a17a1cc0d0b617ee109"
format_stamp: "Formatted at 2026-09-27 21:14:47 on dist-test-slave-r9x1"
server_key: "736494fba9364f5987a9e998ea9464cb"
server_key_iv: "9138116f0306bcddb28e4bf553c35ad0"
server_key_version: "encryptionkey@0"
I20260927 21:14:47.448833 13835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:47.455596 13828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } attempt: 2
I20260927 21:14:47.456027 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.455913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43866)
0927 21:14:47.455966 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.455998 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 21:14:47.456319 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.456242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43870)
0927 21:14:47.456278 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.456301 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 21:14:47.456398 13828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37347: connect: Connection refused (error 111)
I20260927 21:14:47.473093 13835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:47.475251 13835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:47.475394 13835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:47.479151 13835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37347
I20260927 21:14:47.479193 13902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37347 every 8 connection(s)
I20260927 21:14:47.479514 13835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:14:47.480325 13903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:47.481899 13903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:14:47.482673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 13835
I20260927 21:14:47.482780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543529727471-392-0/minicluster-data/master-2/wal/instance
I20260927 21:14:47.482963 392 external_mini_cluster.cc:1428] Setting key 594ebed1831c6573ad83c3b2c0be4ee1
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:14:47.491266 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.482607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.0.98.62:39683)
0927 21:14:47.482660 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.482662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.483004 (+ 342us) server_negotiation.cc:407] Connection header received
0927 21:14:47.483556 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.483557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.483571 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.483617 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.484179 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.484651 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.485612 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.485801 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.485858 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.488054 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.488056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.488056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.488065 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.489694 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.490218 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.490219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.490219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.490281 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.490594 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.490595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.490596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.490727 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.490789 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.491082 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.491168 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:14:47.491485 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.483326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.0.98.60:37347)
0927 21:14:47.483484 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.483490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.483525 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:14:47.483605 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.483606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.483622 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.483691 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.484272 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.485049 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.485893 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.486198 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.486255 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.488509 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.488517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.488517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.488525 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.490182 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.490510 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.490512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.490512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.490562 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.490718 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.490719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.490719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.490940 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.491075 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.491175 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.491304 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260927 21:14:47.491485 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.482623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39683 (local address 127.0.0.1:47954)
0927 21:14:47.482828 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.482910 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:47.483447 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:47.483647 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:47.483651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:47.483937 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:47.484090 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.484095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.484721 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.484722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.485523 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.485526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.485874 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.485875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.485995 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.486461 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:47.486471 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:47.487963 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:47.489789 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.489796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.489798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.490147 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.490378 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.490379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.490379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.490522 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.490811 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:47.490818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:47.490943 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 21:14:47.491051 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 21:14:47.491240 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260927 21:14:47.492079 13903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:47.499862 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.492330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37355 (local address 127.0.0.1:52598)
0927 21:14:47.492445 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.492454 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:47.492487 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:47.492605 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:47.492605 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:47.492738 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:47.492805 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.492807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.493752 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.493753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.494549 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.494552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.494985 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.494986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.495051 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.495617 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:47.495622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:47.496937 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:47.498832 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.498833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.498834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.499005 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.499235 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.499236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.499237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.499306 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.499558 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:47.499559 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:47.499609 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:14:47.499679 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:14:47.499790 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 21:14:47.499882 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.492403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.0.98.61:37355)
0927 21:14:47.492464 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.492466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.492477 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:14:47.492536 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.492537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.492546 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.492589 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.492904 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.493676 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.494640 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.494912 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.494983 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.497037 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.497039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.497039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.497048 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.498744 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.499075 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.499077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.499077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.499145 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.499375 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.499376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.499376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.499493 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.499554 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.499706 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.499790 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:14:47.500653 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.492332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.0.98.62:39683)
0927 21:14:47.492368 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.492370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.492378 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:14:47.492423 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.492423 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.492433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.492485 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.493067 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.493770 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.494719 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.494996 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.495053 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.497685 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.497687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.497687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.497695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.499111 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.499507 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.499508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.499508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.499563 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.499731 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.499732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.499732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.499856 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.500471 (+ 615us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.500511 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.500582 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:14:47.500746 13903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:14:47.502094 13753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } attempt: 2
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:14:47.509150 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.501331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.0.98.61:37355)
0927 21:14:47.501364 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.501366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.501372 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:14:47.501401 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.501402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.501409 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.501447 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.501966 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.502708 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.503601 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.503903 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.503969 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.506580 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.506582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.506583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.506591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.508033 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.508432 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.508433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.508434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.508493 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.508652 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.508652 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.508653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.508771 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.508948 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.508984 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.509058 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:14:47.509925 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.501049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43894)
0927 21:14:47.501157 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.501168 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:47.501219 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:47.501352 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:47.501353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:47.501509 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:47.501558 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.501560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.502364 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.502365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.503040 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.503041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.504281 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.504282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.504335 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.504819 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:47.504823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:47.507125 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:47.508873 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.508874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.508874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.509102 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.509287 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.509288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.509288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.509338 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.509626 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:47.509627 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:47.509677 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:14:47.509740 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:14:47.509824 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 21:14:47.510165 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.502318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43906)
0927 21:14:47.502526 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.502536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:47.502579 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:47.502690 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:47.502690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:47.502841 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:47.502880 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.502882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.503532 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.503533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.504201 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.504203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.504634 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.504635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.504694 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.505020 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:47.505024 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:47.506795 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:47.509116 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.509117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.509117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.509347 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.509528 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.509529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.509529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.509601 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.509860 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:47.509861 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:47.509907 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:14:47.509976 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:14:47.510076 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260927 21:14:47.510242 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.501083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.0.98.60:37347)
0927 21:14:47.501125 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.501126 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.501204 (+ 78us) server_negotiation.cc:407] Connection header received
0927 21:14:47.501289 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.501290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.501299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.501349 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.501622 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.502286 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.503113 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.503453 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.503521 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.507196 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.507197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.507197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.507203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.508787 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.509167 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.509167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.509168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.509211 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.509396 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.509397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.509397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.509571 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.509623 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.510028 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.510158 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":16}
I20260927 21:14:47.510244 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.502371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.0.98.60:37347)
0927 21:14:47.502555 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.502557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.502564 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:14:47.502623 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.502624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.502633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.502688 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.502954 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.503461 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.504268 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.504566 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.504619 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.506870 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.506871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.506872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.506878 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.509030 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.509419 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.509419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.509420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.509474 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.509671 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.509672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.509672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.509793 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.509853 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.509999 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.510101 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":119,"threads_started":1}
I20260927 21:14:47.510726 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:14:47.511827 13753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311: Bootstrap starting.
I20260927 21:14:47.512626 13753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:47.512946 13753 log.cc:824] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:47.514251 13903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109: Bootstrap starting.
I20260927 21:14:47.515055 13903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:47.515441 13903 log.cc:824] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:47.516268 13753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311: No bootstrap required, opened a new log
I20260927 21:14:47.518394 13753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } }
I20260927 21:14:47.518548 13753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:47.518579 13753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3202f2bbce4520b8d18d376b2b3311, State: Initialized, Role: FOLLOWER
I20260927 21:14:47.518705 13753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } }
I20260927 21:14:47.518977 13918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.519042 13753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:14:47.519724 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.512309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.0.98.60:37347)
0927 21:14:47.512350 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.512352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.512359 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:14:47.512386 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.512387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.512397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.512448 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.513292 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.513961 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.515037 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.515320 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.515363 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.516798 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.516799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.516799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.516807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.518152 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.518709 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.518710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.518710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.518768 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.518995 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.518996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.518996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.519294 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.519360 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.519556 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.519631 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:14:47.519724 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.511947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.0.98.62:39683)
0927 21:14:47.511986 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.511988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.512147 (+ 159us) server_negotiation.cc:407] Connection header received
0927 21:14:47.512346 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.512347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.512357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.512401 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.513219 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.513989 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.515075 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.515324 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.515376 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.516893 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.516894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.516895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.516901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.518387 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.518695 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.518697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.518697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.518770 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.518989 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.518990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.518991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.519122 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.519224 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.519554 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.519630 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:14:47.520582 13918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:14:47.520924 13903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109: No bootstrap required, opened a new log
I20260927 21:14:47.522976 13903 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } }
I20260927 21:14:47.523149 13903 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:47.523182 13903 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec676f0a35774a17a1cc0d0b617ee109, State: Initialized, Role: FOLLOWER
I20260927 21:14:47.523313 13903 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } }
I20260927 21:14:47.523599 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.523685 13903 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:14:47.523689 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:14:47.521906 13685 master_runner.cc:186] Error getting master registration for 127.0.98.61:37355: OK, instance_id { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" instance_seqno: 1790543687311245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:14:47.524799 13685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 21:14:47.525750 13929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:14:47.525801 13929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:14:47.526546 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.512261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.0.98.61:37355)
0927 21:14:47.512449 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.512451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.512458 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:14:47.512475 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.512476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.512484 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.512526 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.513220 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.513989 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.515092 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.515507 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.515546 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.520268 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.520270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.520270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.520279 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.521848 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.525312 (+ 3464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.525313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.525314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.525408 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.525652 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.525653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.525653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.525794 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.525877 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.526372 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.526459 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
W20260927 21:14:47.527334 13941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:14:47.527385 13941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:14:47.527086 13835 master_runner.cc:186] Error getting master registration for 127.0.98.61:37355: OK, instance_id { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" instance_seqno: 1790543687311245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:14:47.529407 13835 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:14:47.547564 13828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } attempt: 3
Sep 27 21:14:47 dist-test-slave-r9x1 krb5kdc[13672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:14:47.556310 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.547992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.0.98.60:37347)
0927 21:14:47.548032 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:47.548034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:47.548042 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:14:47.548066 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:47.548066 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:47.548076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:47.548135 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:47.548492 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.549328 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.550332 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.550657 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.550732 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.553011 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:47.553012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:47.553013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:47.553022 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:47.554650 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.555454 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.555455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.555456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.555537 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:47.555747 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:47.555748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:47.555748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:47.555903 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:47.555976 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:47.556140 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 21:14:47.556223 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:14:47.556322 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:47.547874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37347 (local address 127.0.0.1:43910)
0927 21:14:47.547947 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:47.547960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:47.548002 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:47.548148 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:47.548148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:47.548341 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:47.548411 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.548413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.549411 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.549412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.550222 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:47.550225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:47.550749 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:47.550750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:47.550808 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:47.551321 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:47.551326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:47.552917 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:47.554740 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.554741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.554741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.554959 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.555612 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:47.555613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:47.555613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:47.555678 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:47.555974 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:47.555975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:47.556027 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 21:14:47.556100 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:14:47.556215 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 21:14:47.558382 13828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500: Bootstrap starting.
I20260927 21:14:47.559181 13828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:47.559490 13828 log.cc:824] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:47.560362 13828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500: No bootstrap required, opened a new log
I20260927 21:14:47.562940 13828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } }
I20260927 21:14:47.563097 13828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:47.563131 13828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6ca4d89f8e40bb9eb512e55bdfd500, State: Initialized, Role: FOLLOWER
I20260927 21:14:47.563308 13828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } }
I20260927 21:14:47.563657 13828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:14:47.565603 13956 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:14:47.565670 13956 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:14:47.567795 13760 master_runner.cc:263] No leader master found from master 4f6ca4d89f8e40bb9eb512e55bdfd500
I20260927 21:14:47.567875 13760 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:14:47.563620 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.568079 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:14:47.657138 13706 tablet.cc:2369] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:14:47.726076 13856 tablet.cc:2369] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:14:47.742100 13918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:14:47.742225 13918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } }
I20260927 21:14:47.742491 13918 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f6ca4d89f8e40bb9eb512e55bdfd500 (127.0.98.61:37355), ec676f0a35774a17a1cc0d0b617ee109 (127.0.98.60:37347)
I20260927 21:14:47.742833 13877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e3202f2bbce4520b8d18d376b2b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec676f0a35774a17a1cc0d0b617ee109" is_pre_election: true
I20260927 21:14:47.742835 13802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e3202f2bbce4520b8d18d376b2b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" is_pre_election: true
I20260927 21:14:47.742973 13802 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e3202f2bbce4520b8d18d376b2b3311 in term 0.
I20260927 21:14:47.742986 13877 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e3202f2bbce4520b8d18d376b2b3311 in term 0.
I20260927 21:14:47.743162 13703 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e3202f2bbce4520b8d18d376b2b3311, ec676f0a35774a17a1cc0d0b617ee109; no voters:
I20260927 21:14:47.743343 13918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:14:47.743427 13918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:14:47.743445 13918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:47.744210 13918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } }
I20260927 21:14:47.744329 13918 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [CANDIDATE]: Term 1 election: Requested vote from peers 4f6ca4d89f8e40bb9eb512e55bdfd500 (127.0.98.61:37355), ec676f0a35774a17a1cc0d0b617ee109 (127.0.98.60:37347)
I20260927 21:14:47.744532 13802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e3202f2bbce4520b8d18d376b2b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500"
I20260927 21:14:47.744577 13877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e3202f2bbce4520b8d18d376b2b3311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec676f0a35774a17a1cc0d0b617ee109"
I20260927 21:14:47.744623 13802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:47.744661 13877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:47.745353 13802 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e3202f2bbce4520b8d18d376b2b3311 in term 1.
I20260927 21:14:47.745437 13877 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e3202f2bbce4520b8d18d376b2b3311 in term 1.
I20260927 21:14:47.745566 13703 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e3202f2bbce4520b8d18d376b2b3311, 4f6ca4d89f8e40bb9eb512e55bdfd500; no voters:
I20260927 21:14:47.745658 13918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:14:47.745791 13918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [term 1 LEADER]: Becoming Leader. State: Replica: 3e3202f2bbce4520b8d18d376b2b3311, State: Running, Role: LEADER
I20260927 21:14:47.745893 13918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } }
I20260927 21:14:47.746302 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3202f2bbce4520b8d18d376b2b3311. Latest consensus state: current_term: 1 leader_uuid: "3e3202f2bbce4520b8d18d376b2b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.746414 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:47.746606 13960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:14:47.746783 13960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:14:47.748118 13877 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [term 1 FOLLOWER]: Refusing update from remote peer 3e3202f2bbce4520b8d18d376b2b3311: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:14:47.748148 13802 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [term 1 FOLLOWER]: Refusing update from remote peer 3e3202f2bbce4520b8d18d376b2b3311: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:14:47.748405 13958 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:14:47.748512 13918 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:14:47.749598 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3202f2bbce4520b8d18d376b2b3311. Latest consensus state: current_term: 1 leader_uuid: "3e3202f2bbce4520b8d18d376b2b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.749699 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:14:47.749747 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3202f2bbce4520b8d18d376b2b3311. Latest consensus state: current_term: 1 leader_uuid: "3e3202f2bbce4520b8d18d376b2b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.749835 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:14:47.750679 13918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e3202f2bbce4520b8d18d376b2b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.750726 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e3202f2bbce4520b8d18d376b2b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.750788 13918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:47.750809 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:47.750855 13962 mvcc.cc:205] Tried to move back new op lower bound from 7334066945014067200 to 7334066945007882240. Current Snapshot: MvccSnapshot[applied={T|T < 7334066945014067200}]
I20260927 21:14:47.750993 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e3202f2bbce4520b8d18d376b2b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.751065 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:14:47.751120 13960 catalog_manager.cc:1347] Generated new cluster ID: 2dcee973e9e1411993ffcfd623335f3a
I20260927 21:14:47.751157 13960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:14:47.751410 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e3202f2bbce4520b8d18d376b2b3311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3202f2bbce4520b8d18d376b2b3311" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } peers { permanent_uuid: "4f6ca4d89f8e40bb9eb512e55bdfd500" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37355 } } peers { permanent_uuid: "ec676f0a35774a17a1cc0d0b617ee109" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37347 } } }
I20260927 21:14:47.751487 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:14:47.766286 13960 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:14:47.767032 13960 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:14:47.778349 13960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e3202f2bbce4520b8d18d376b2b3311: Generated new TSK 0
I20260927 21:14:47.778542 13960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:14:48.527817 13941 catalog_manager.cc:1259] Loaded cluster ID: 2dcee973e9e1411993ffcfd623335f3a
I20260927 21:14:48.527894 13941 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109: loading cluster ID for follower catalog manager: success
I20260927 21:14:48.528710 13941 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109: acquiring CA information for follower catalog manager: success
I20260927 21:14:48.529027 13941 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec676f0a35774a17a1cc0d0b617ee109: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:14:48.566097 13956 catalog_manager.cc:1259] Loaded cluster ID: 2dcee973e9e1411993ffcfd623335f3a
I20260927 21:14:48.566185 13956 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500: loading cluster ID for follower catalog manager: success
I20260927 21:14:48.566982 13956 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500: acquiring CA information for follower catalog manager: success
I20260927 21:14:48.567266 13956 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f6ca4d89f8e40bb9eb512e55bdfd500: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:14:48.849486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 13685
I20260927 21:14:48.854485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 13760
I20260927 21:14:48.858887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 13835
2026-09-27T21:14:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2901 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:14:48 dist-test-slave-r9x1 krb5kdc[13975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:14:48 dist-test-slave-r9x1 krb5kdc[13975](info): set up 2 sockets
Sep 27 21:14:48 dist-test-slave-r9x1 krb5kdc[13975](info): commencing operation
krb5kdc: starting...
W20260927 21:14:49.825342 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:14:49 dist-test-slave-r9x1 krb5kdc[13975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:14:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:14:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:14:49.945022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38623
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:50.028079 13988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:50.028308 13988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:50.028332 13988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:50.029539 13988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:14:50.029580 13988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:50.029593 13988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:50.029605 13988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:14:50.029616 13988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:14:50.031121 13988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:50.031344 13988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:50.031524 13988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:50.036258 13996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:50.036300 13993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:50.036348 13994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:50.036379 13988 server_base.cc:1034] running on GCE node
I20260927 21:14:50.036713 13988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:50.037021 13988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:50.038172 13988 hybrid_clock.cc:648] HybridClock initialized: now 1790543690038153 us; error 37 us; skew 500 ppm
Sep 27 21:14:50 dist-test-slave-r9x1 krb5kdc[13975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:50.040614 13988 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:14:50.041085 13988 webserver.cc:460] Webserver started at http://127.0.98.62:36467/ using document root <none> and password file <none>
I20260927 21:14:50.041337 13988 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:50.041389 13988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:50.041523 13988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:14:50.042609 13988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "af30e0796238460ea9b85a8787f77cf8"
format_stamp: "Formatted at 2026-09-27 21:14:50 on dist-test-slave-r9x1"
server_key: "15068c97f7aef7890a326283b83c0c8f"
server_key_iv: "f316936c01e8aac7ca1cd389bff8995d"
server_key_version: "encryptionkey@0"
I20260927 21:14:50.042954 13988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "af30e0796238460ea9b85a8787f77cf8"
format_stamp: "Formatted at 2026-09-27 21:14:50 on dist-test-slave-r9x1"
server_key: "15068c97f7aef7890a326283b83c0c8f"
server_key_iv: "f316936c01e8aac7ca1cd389bff8995d"
server_key_version: "encryptionkey@0"
I20260927 21:14:50.044340 13988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:14:50.045188 14003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:50.045400 13988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:14:50.045491 13988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "af30e0796238460ea9b85a8787f77cf8"
format_stamp: "Formatted at 2026-09-27 21:14:50 on dist-test-slave-r9x1"
server_key: "15068c97f7aef7890a326283b83c0c8f"
server_key_iv: "f316936c01e8aac7ca1cd389bff8995d"
server_key_version: "encryptionkey@0"
I20260927 21:14:50.045565 13988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:50.076582 13988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:50.078748 13988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:50.078895 13988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:50.082866 13988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38623
I20260927 21:14:50.082962 14055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38623 every 8 connection(s)
I20260927 21:14:50.083271 13988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:14:50.084179 14056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:50.086790 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8: Bootstrap starting.
I20260927 21:14:50.087393 14056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:50.087658 14056 log.cc:824] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:50.088512 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8: No bootstrap required, opened a new log
I20260927 21:14:50.090263 14056 raft_consensus.cc:348] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af30e0796238460ea9b85a8787f77cf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } }
I20260927 21:14:50.090423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 13988
I20260927 21:14:50.090476 14056 raft_consensus.cc:374] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:50.090508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:14:50.090513 14056 raft_consensus.cc:729] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af30e0796238460ea9b85a8787f77cf8, State: Initialized, Role: FOLLOWER
I20260927 21:14:50.090714 392 external_mini_cluster.cc:1428] Setting key 3f2ca6bddd84dda3201848a9921626a5
I20260927 21:14:50.090679 14056 consensus_queue.cc:260] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af30e0796238460ea9b85a8787f77cf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } }
I20260927 21:14:50.090756 14056 raft_consensus.cc:388] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:14:50.090787 14056 raft_consensus.cc:482] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:14:50.090837 14056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:50.091583 14056 raft_consensus.cc:504] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af30e0796238460ea9b85a8787f77cf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } }
I20260927 21:14:50.091691 14056 leader_election.cc:302] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af30e0796238460ea9b85a8787f77cf8; no voters:
I20260927 21:14:50.091853 14056 leader_election.cc:288] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:14:50.091883 14061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:14:50.091985 14061 raft_consensus.cc:686] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [term 1 LEADER]: Becoming Leader. State: Replica: af30e0796238460ea9b85a8787f77cf8, State: Running, Role: LEADER
I20260927 21:14:50.092129 14056 sys_catalog.cc:564] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:14:50.092136 14061 consensus_queue.cc:237] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af30e0796238460ea9b85a8787f77cf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } }
I20260927 21:14:50.092629 14062 sys_catalog.cc:455] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af30e0796238460ea9b85a8787f77cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af30e0796238460ea9b85a8787f77cf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } } }
I20260927 21:14:50.092718 14062 sys_catalog.cc:458] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:50.092907 14070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:14:50.092880 14063 sys_catalog.cc:455] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af30e0796238460ea9b85a8787f77cf8. Latest consensus state: current_term: 1 leader_uuid: "af30e0796238460ea9b85a8787f77cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af30e0796238460ea9b85a8787f77cf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38623 } } }
I20260927 21:14:50.092971 14063 sys_catalog.cc:458] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:50.093531 14070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:14:50.095815 14070 catalog_manager.cc:1347] Generated new cluster ID: 39963e83c3234ccea4e2122a5c7b1540
I20260927 21:14:50.095875 14070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:14:50 dist-test-slave-r9x1 krb5kdc[13975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:50.102144 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:50.091229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.0.98.62:38623)
0927 21:14:50.091680 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:50.091692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:50.091730 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:14:50.091859 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:50.091865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:50.091919 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:50.092806 (+ 887us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:50.092927 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:50.094534 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:50.095311 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:50.095854 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:50.095955 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:50.098383 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:50.098388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:50.098388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:50.098402 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:50.100362 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:50.100757 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:50.100760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:50.100760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:50.100893 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:50.101073 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:50.101074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:50.101074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:50.101457 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:50.101657 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:50.101740 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 21:14:50.101914 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":351,"threads_started":1}
I20260927 21:14:50.103273 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:14:50.117949 14070 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:14:50.118515 14070 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:14:50.123507 14070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af30e0796238460ea9b85a8787f77cf8: Generated new TSK 0
I20260927 21:14:50.123749 14070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:14:50.157428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 13988
2026-09-27T21:14:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1297 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:14:50 dist-test-slave-r9x1 krb5kdc[14083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:14:50 dist-test-slave-r9x1 krb5kdc[14083](info): set up 2 sockets
Sep 27 21:14:50 dist-test-slave-r9x1 krb5kdc[14083](info): commencing operation
krb5kdc: starting...
W20260927 21:14:51.113242 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:14:51 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:14:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:14:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:14:51.232295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33899
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:51.313325 14096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:51.313660 14096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:51.313709 14096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:51.315042 14096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:14:51.315099 14096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:51.315114 14096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:51.315125 14096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:14:51.315136 14096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:14:51.316841 14096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:51.317085 14096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:51.317317 14096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:51.322173 14102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:51.322252 14104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:51.322253 14101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:51.322458 14096 server_base.cc:1034] running on GCE node
I20260927 21:14:51.322703 14096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:51.323065 14096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:51.324347 14096 hybrid_clock.cc:648] HybridClock initialized: now 1790543691324324 us; error 33 us; skew 500 ppm
Sep 27 21:14:51 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:51.332861 14096 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:14:51.333401 14096 webserver.cc:460] Webserver started at http://127.0.98.62:36317/ using document root <none> and password file <none>
I20260927 21:14:51.333613 14096 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:51.333665 14096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:51.333807 14096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:14:51.334910 14096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "57984bfd275a48d48bda0f10d8121810"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "cd31874de69eb6871c3bd14663de5518"
server_key_iv: "01990b88c71d5530da2849d31246af6f"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.335263 14096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "57984bfd275a48d48bda0f10d8121810"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "cd31874de69eb6871c3bd14663de5518"
server_key_iv: "01990b88c71d5530da2849d31246af6f"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.336655 14096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:14:51.337741 14111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.338088 14096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:51.338191 14096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "57984bfd275a48d48bda0f10d8121810"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "cd31874de69eb6871c3bd14663de5518"
server_key_iv: "01990b88c71d5530da2849d31246af6f"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.338261 14096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:51.353158 14096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:51.355376 14096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:51.355528 14096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:51.359536 14096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33899
I20260927 21:14:51.359578 14163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33899 every 8 connection(s)
I20260927 21:14:51.359933 14096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:14:51.361387 14164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:51.363814 14164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Bootstrap starting.
I20260927 21:14:51.364409 14164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:51.364643 14164 log.cc:824] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:51.365468 14164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: No bootstrap required, opened a new log
I20260927 21:14:51.367210 14164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:51.367388 14164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:51.367376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 14096
I20260927 21:14:51.367413 14164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57984bfd275a48d48bda0f10d8121810, State: Initialized, Role: FOLLOWER
I20260927 21:14:51.367473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:14:51.367590 14164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:51.367656 392 external_mini_cluster.cc:1428] Setting key e71bad67ccb49cad3611fb6c49f47f32
I20260927 21:14:51.367669 14164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:14:51.367697 14164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:14:51.367744 14164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:51.368537 14164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:51.368649 14164 leader_election.cc:302] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57984bfd275a48d48bda0f10d8121810; no voters:
I20260927 21:14:51.368829 14164 leader_election.cc:288] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:14:51.368842 14169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:14:51.368974 14169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 1 LEADER]: Becoming Leader. State: Replica: 57984bfd275a48d48bda0f10d8121810, State: Running, Role: LEADER
I20260927 21:14:51.369124 14164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:14:51.369124 14169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:51.369725 14170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57984bfd275a48d48bda0f10d8121810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } } }
I20260927 21:14:51.369817 14170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:51.370048 14171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57984bfd275a48d48bda0f10d8121810. Latest consensus state: current_term: 1 leader_uuid: "57984bfd275a48d48bda0f10d8121810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } } }
I20260927 21:14:51.370126 14171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:51.370122 14178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:14:51.370756 14178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:14:51.372572 14178 catalog_manager.cc:1347] Generated new cluster ID: f06d2409478b49b39ef35ef4cd38d36d
I20260927 21:14:51.372637 14178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:14:51 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:51.378569 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.368164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.0.98.62:33899)
0927 21:14:51.368649 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.368660 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.368696 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:14:51.368818 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.368823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.368877 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.369771 (+ 894us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:51.369871 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.371258 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.372248 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.372744 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.372843 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.375320 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:51.375326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:51.375326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:51.375342 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:51.376965 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.377350 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.377355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.377355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.377463 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.377657 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.377658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.377658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.377936 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:51.378100 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.378212 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.378375 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":386,"threads_started":1}
I20260927 21:14:51.389788 14178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:14:51.390395 14178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:14:51.393505 14178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Generated new TSK 0
I20260927 21:14:51.393705 14178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:14:51.428653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33899
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:51.517089 14190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:51.517356 14190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:51.517388 14190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:51.518688 14190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:51.518728 14190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:51.518755 14190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:14:51.520323 14190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:51.520561 14190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:51.520781 14190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:51.525691 14196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:51.525712 14195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:51.525691 14198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:51.525908 14190 server_base.cc:1034] running on GCE node
I20260927 21:14:51.526137 14190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:51.526362 14190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:51.527501 14190 hybrid_clock.cc:648] HybridClock initialized: now 1790543691527498 us; error 33 us; skew 500 ppm
Sep 27 21:14:51 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:51.530176 14190 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:14:51.530618 14190 webserver.cc:460] Webserver started at http://127.0.98.1:36363/ using document root <none> and password file <none>
I20260927 21:14:51.530799 14190 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:51.530833 14190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:51.530926 14190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:14:51.532045 14190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "14859606b1ac4cf280ba942361bf0fa0"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "2dffac691a3cb07616307d47b3b60fdb"
server_key_iv: "7f030ad0ce8c726e879414ffc9e0fcbf"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.532385 14190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "14859606b1ac4cf280ba942361bf0fa0"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "2dffac691a3cb07616307d47b3b60fdb"
server_key_iv: "7f030ad0ce8c726e879414ffc9e0fcbf"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.533747 14190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:14:51.534557 14205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.534740 14190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:14:51.534824 14190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "14859606b1ac4cf280ba942361bf0fa0"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "2dffac691a3cb07616307d47b3b60fdb"
server_key_iv: "7f030ad0ce8c726e879414ffc9e0fcbf"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.534883 14190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:51.544334 14190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:51.546480 14190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:51.546625 14190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:51.546859 14190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:51.547147 14190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:51.547179 14190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.547204 14190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:51.547226 14190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.552631 14190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44677
I20260927 21:14:51.552702 14318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44677 every 8 connection(s)
I20260927 21:14:51.553002 14190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:14:51.554207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 14190
I20260927 21:14:51.554317 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:14:51.554518 392 external_mini_cluster.cc:1428] Setting key 07d5864330169a5c3c1a576d999c25f1
Sep 27 21:14:51 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 21:14:51.563467 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.553882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39343 (local address 127.0.98.62:33899)
0927 21:14:51.553937 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.553939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.554199 (+ 260us) server_negotiation.cc:407] Connection header received
0927 21:14:51.554829 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.554831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.554844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.554886 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:51.556077 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.556624 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.558086 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.558286 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.558360 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.560654 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:51.560656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:51.560656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:51.560664 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:51.562117 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.562533 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.562535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.562535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.562589 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.562829 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.562830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.562830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.562960 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:51.563015 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.563284 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.563373 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:14:51.563657 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.553854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:39343)
0927 21:14:51.554051 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:51.554101 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:51.554731 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:51.554972 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:51.554976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:51.555674 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:51.555964 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.555971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.556721 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.556722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.557980 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.557984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.558376 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.558377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.558521 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.559043 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:51.559054 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:51.560551 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:51.562203 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:51.562213 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:51.562215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:51.562462 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:51.562646 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:51.562647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:51.562647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:51.562772 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:51.563015 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:51.563022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:51.563166 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 21:14:51.563255 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 21:14:51.563434 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
I20260927 21:14:51.564157 14319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33899
I20260927 21:14:51.564254 14319 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:51.564409 14319 heartbeater.cc:511] Master 127.0.98.62:33899 requested a full tablet report, sending...
I20260927 21:14:51.564918 14128 ts_manager.cc:194] Registered new tserver with Master: 14859606b1ac4cf280ba942361bf0fa0 (127.0.98.1:44677)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 21:14:51.565685 14128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:14:51.584115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33899
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:51.678539 14324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:51.678774 14324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:51.678800 14324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:51.680104 14324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:51.680166 14324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:51.680197 14324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:14:51.681898 14324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:51.682175 14324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:51.682425 14324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:51.687391 14332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:51.687391 14329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:51.687454 14330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:51.687980 14324 server_base.cc:1034] running on GCE node
I20260927 21:14:51.688205 14324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:51.688450 14324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:51.689606 14324 hybrid_clock.cc:648] HybridClock initialized: now 1790543691689583 us; error 34 us; skew 500 ppm
Sep 27 21:14:51 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:51.692206 14324 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:14:51.692655 14324 webserver.cc:460] Webserver started at http://127.0.98.2:37475/ using document root <none> and password file <none>
I20260927 21:14:51.692888 14324 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:51.692939 14324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:51.693071 14324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:14:51.694227 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5f74a8a0d484781807632e23bdea3eb"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "07b019ac1b512126c8f5f081f1ad1584"
server_key_iv: "2435e2161c306ff0aff590ed53fbf963"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.694586 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5f74a8a0d484781807632e23bdea3eb"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "07b019ac1b512126c8f5f081f1ad1584"
server_key_iv: "2435e2161c306ff0aff590ed53fbf963"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.695941 14324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:14:51.696688 14339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.696837 14324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:14:51.696959 14324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "f5f74a8a0d484781807632e23bdea3eb"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "07b019ac1b512126c8f5f081f1ad1584"
server_key_iv: "2435e2161c306ff0aff590ed53fbf963"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.697034 14324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:51.722317 14324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:51.724545 14324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:51.724694 14324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:51.724925 14324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:51.725232 14324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:51.725319 14324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.725355 14324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:51.725373 14324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.730995 14324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44295
I20260927 21:14:51.731076 14452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44295 every 8 connection(s)
I20260927 21:14:51.731374 14324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:14:51 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:51.740188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 14324
I20260927 21:14:51.740335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:14:51.740592 392 external_mini_cluster.cc:1428] Setting key 2d9a3386317b0b0ce2dfdaabdb873fae
I20260927 21:14:51.741123 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.732125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42437 (local address 127.0.98.62:33899)
0927 21:14:51.732194 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.732196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.732386 (+ 190us) server_negotiation.cc:407] Connection header received
0927 21:14:51.732933 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.732935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.732947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.733000 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:51.733996 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.734498 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.735498 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.735711 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.735768 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.738026 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:51.738028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:51.738028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:51.738037 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:51.739385 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.739855 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.739856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.739857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.739920 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.740315 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.740317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.740317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.740468 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:51.740532 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.740968 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.741057 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:14:51.741390 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.732085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:42437)
0927 21:14:51.732257 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:51.732295 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:51.732816 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:51.733074 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:51.733078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:51.733693 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:51.733903 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.733909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.734593 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.734594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.735411 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.735413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.735786 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.735787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.735915 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.736402 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:51.736411 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:51.737906 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:51.739486 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:51.739496 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:51.739499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:51.739765 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:51.740026 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:51.740028 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:51.740028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:51.740215 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:51.740585 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:51.740594 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:51.740819 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 21:14:51.740923 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:14:51.741111 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260927 21:14:51.741801 14453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33899
I20260927 21:14:51.741905 14453 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:51.742091 14453 heartbeater.cc:511] Master 127.0.98.62:33899 requested a full tablet report, sending...
I20260927 21:14:51.742436 14128 ts_manager.cc:194] Registered new tserver with Master: f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295)
I20260927 21:14:51.742945 14128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42437
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:14:51.767702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33899
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:51.847704 14458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:51.847926 14458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:51.847958 14458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:51.849182 14458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:51.849228 14458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:51.849288 14458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:14:51.851199 14458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:51.851480 14458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:51.851706 14458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:51.856570 14466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:51.856761 14458 server_base.cc:1034] running on GCE node
W20260927 21:14:51.856570 14463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:51.856573 14464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:51.857050 14458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:51.857354 14458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:51.858513 14458 hybrid_clock.cc:648] HybridClock initialized: now 1790543691858502 us; error 28 us; skew 500 ppm
Sep 27 21:14:51 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:51.860996 14458 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:14:51.861507 14458 webserver.cc:460] Webserver started at http://127.0.98.3:35475/ using document root <none> and password file <none>
I20260927 21:14:51.861704 14458 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:51.861743 14458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:51.861837 14458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:14:51.862844 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "47af539ed94c47eba5b6bd07687464b6"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "c787cff65469c395d19c57ef294b4237"
server_key_iv: "cc33550399d9297903ece970f18623e6"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.863133 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47af539ed94c47eba5b6bd07687464b6"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "c787cff65469c395d19c57ef294b4237"
server_key_iv: "cc33550399d9297903ece970f18623e6"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.864398 14458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:14:51.865099 14473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.865334 14458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:51.865419 14458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "47af539ed94c47eba5b6bd07687464b6"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "c787cff65469c395d19c57ef294b4237"
server_key_iv: "cc33550399d9297903ece970f18623e6"
server_key_version: "encryptionkey@0"
I20260927 21:14:51.865475 14458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:51.881711 14458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:51.883661 14458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:51.883795 14458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:51.884056 14458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:51.884359 14458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:51.884397 14458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.884423 14458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:51.884433 14458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:51.890322 14458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45989
I20260927 21:14:51.890358 14586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45989 every 8 connection(s)
I20260927 21:14:51.890671 14458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:14:51.892746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 14458
I20260927 21:14:51.892827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:14:51.892992 392 external_mini_cluster.cc:1428] Setting key edade5dc7e43e9bffbb67dc50361681d
Sep 27 21:14:51 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:51.899901 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.891401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51983 (local address 127.0.98.62:33899)
0927 21:14:51.891480 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.891482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.891609 (+ 127us) server_negotiation.cc:407] Connection header received
0927 21:14:51.892115 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.892117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.892129 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.892180 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:51.893190 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.893886 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.894914 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.895103 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.895159 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.897333 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:51.897335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:51.897335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:51.897343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:51.898573 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.898996 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.898997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.898998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.899057 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.899273 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.899274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.899274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.899393 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:51.899455 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.899724 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.899806 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:14:51.900116 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.891324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:51983)
0927 21:14:51.891491 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:51.891533 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:51.892017 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:51.892245 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:51.892248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:51.892868 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:51.893105 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.893109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.893974 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.893975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.894826 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.894829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.895181 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.895181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.895302 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.895834 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:51.895843 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:51.897205 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:51.898672 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:51.898680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:51.898682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:51.898924 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:51.899114 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:51.899115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:51.899116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:51.899220 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:51.899455 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:51.899461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:51.899602 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 21:14:51.899692 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:14:51.899883 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260927 21:14:51.900553 14587 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33899
I20260927 21:14:51.900660 14587 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:51.900851 14587 heartbeater.cc:511] Master 127.0.98.62:33899 requested a full tablet report, sending...
I20260927 21:14:51.901244 14128 ts_manager.cc:194] Registered new tserver with Master: 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989)
I20260927 21:14:51.901635 14128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51983
I20260927 21:14:51.904624 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:14:51.911412 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.905930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.0.98.62:33899)
0927 21:14:51.905988 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.905990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.906127 (+ 137us) server_negotiation.cc:407] Connection header received
0927 21:14:51.906152 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.906153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.906163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.906203 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:51.906717 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.907192 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.907880 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.908055 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.908101 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.909368 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:51.909369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:51.909370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:51.909377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:51.910444 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.910759 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.910760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.910760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.910809 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:51.910951 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:51.910952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:51.910952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:51.911098 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:51.911173 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.911277 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.911336 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:14:51.912946 14128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:14:51.914582 14128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:14:51.920821 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.916910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.0.98.2:44295)
0927 21:14:51.917087 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.917091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.917099 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:14:51.917129 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.917131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.917142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.917210 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:51.917546 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.918373 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.919694 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.920415 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.920485 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.920544 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.920692 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.920738 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260927 21:14:51.920821 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.916667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44295 (local address 127.0.0.1:50020)
0927 21:14:51.916842 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:51.916876 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:51.916941 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:51.917238 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:51.917242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:51.917244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:51.917338 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.917341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.918462 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.918463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.919623 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.919626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.920498 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.920499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.920592 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.920601 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:14:51.920670 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:14:51.920726 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":24,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260927 21:14:51.921084 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.917101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.0.98.1:44677)
0927 21:14:51.917253 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.917256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.917288 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:14:51.917324 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.917325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.917339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.917414 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:51.917588 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.918735 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.919943 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.920429 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.920862 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.920932 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.920960 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.921006 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260927 21:14:51.920821 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.916667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44677 (local address 127.0.0.1:35814)
0927 21:14:51.916837 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:51.916876 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:51.916941 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:51.917415 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:51.917416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:51.917418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:51.917473 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.917474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.918837 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.918838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.919861 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.919863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.920520 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.920521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.920583 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.920601 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 21:14:51.920647 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:14:51.920720 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260927 21:14:51.921461 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.916915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.0.98.3:45989)
0927 21:14:51.917284 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.917289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.917467 (+ 178us) server_negotiation.cc:407] Connection header received
0927 21:14:51.917508 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.917511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.917525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.917594 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:51.917770 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.918807 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.919967 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.920681 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.920739 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.920789 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.921338 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.921382 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":86,"threads_started":1}
I20260927 21:14:51.921242 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.916672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45989 (local address 127.0.0.1:51802)
0927 21:14:51.917337 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:51.917346 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:51.917414 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:51.917671 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:51.917671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:51.917673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:51.917709 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.917710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.918881 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.918882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.919878 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.919880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.920822 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.920823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.920881 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.920886 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:14:51.921133 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 21:14:51.921177 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":90,"threads_started":1}
I20260927 21:14:51.921746 14387 tablet_service.cc:1467] Processing CreateTablet for tablet 244231396ac14d1b99727bfacb136770 (DEFAULT_TABLE table=test-table [id=e54ec078f72740c6a0216563c30e8648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:14:51.922012 14253 tablet_service.cc:1467] Processing CreateTablet for tablet 244231396ac14d1b99727bfacb136770 (DEFAULT_TABLE table=test-table [id=e54ec078f72740c6a0216563c30e8648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:14:51.922128 14387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244231396ac14d1b99727bfacb136770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:51.922116 14521 tablet_service.cc:1467] Processing CreateTablet for tablet 244231396ac14d1b99727bfacb136770 (DEFAULT_TABLE table=test-table [id=e54ec078f72740c6a0216563c30e8648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:14:51.922313 14253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244231396ac14d1b99727bfacb136770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:51.922449 14521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244231396ac14d1b99727bfacb136770. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:51.925242 14604 tablet_bootstrap.cc:492] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: Bootstrap starting.
I20260927 21:14:51.925729 14606 tablet_bootstrap.cc:492] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Bootstrap starting.
I20260927 21:14:51.925915 14604 tablet_bootstrap.cc:654] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:51.926357 14606 tablet_bootstrap.cc:654] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:51.926465 14604 log.cc:824] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:51.926796 14606 log.cc:824] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:51.927126 14604 tablet_bootstrap.cc:492] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: No bootstrap required, opened a new log
I20260927 21:14:51.927112 14605 tablet_bootstrap.cc:492] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: Bootstrap starting.
I20260927 21:14:51.927191 14604 ts_tablet_manager.cc:1397] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:14:51.927636 14605 tablet_bootstrap.cc:654] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:51.928092 14605 log.cc:824] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:51.928556 14606 tablet_bootstrap.cc:492] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: No bootstrap required, opened a new log
I20260927 21:14:51.928628 14606 ts_tablet_manager.cc:1397] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:14:51.928787 14605 tablet_bootstrap.cc:492] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: No bootstrap required, opened a new log
I20260927 21:14:51.928859 14605 ts_tablet_manager.cc:1397] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:14:51.928926 14604 raft_consensus.cc:348] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:51.929082 14604 raft_consensus.cc:374] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:51.929137 14604 raft_consensus.cc:729] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47af539ed94c47eba5b6bd07687464b6, State: Initialized, Role: FOLLOWER
I20260927 21:14:51.929298 14604 consensus_queue.cc:260] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:51.929565 14604 ts_tablet_manager.cc:1428] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 21:14:51.929693 14587 heartbeater.cc:503] Master 127.0.98.62:33899 was elected leader, sending a full tablet report...
I20260927 21:14:51.930609 14606 raft_consensus.cc:348] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:51.930794 14606 raft_consensus.cc:374] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:51.930837 14606 raft_consensus.cc:729] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14859606b1ac4cf280ba942361bf0fa0, State: Initialized, Role: FOLLOWER
I20260927 21:14:51.930954 14606 consensus_queue.cc:260] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:51.931237 14606 ts_tablet_manager.cc:1428] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:14:51.931128 14605 raft_consensus.cc:348] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:51.931257 14319 heartbeater.cc:503] Master 127.0.98.62:33899 was elected leader, sending a full tablet report...
I20260927 21:14:51.931280 14605 raft_consensus.cc:374] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:51.931358 14605 raft_consensus.cc:729] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f74a8a0d484781807632e23bdea3eb, State: Initialized, Role: FOLLOWER
I20260927 21:14:51.931500 14605 consensus_queue.cc:260] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:51.931754 14605 ts_tablet_manager.cc:1428] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:14:51.931757 14453 heartbeater.cc:503] Master 127.0.98.62:33899 was elected leader, sending a full tablet report...
I20260927 21:14:51.957528 14611 raft_consensus.cc:482] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:14:51.957655 14611 raft_consensus.cc:504] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:51.958026 14611 leader_election.cc:288] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989), f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295)
I20260927 21:14:51.960930 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.958317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41023 (local address 127.0.98.2:44295)
0927 21:14:51.958364 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.958366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.958377 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:14:51.958421 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.958422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.958436 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.958505 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:51.958660 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.959183 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.960200 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.960591 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.960647 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.960659 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.960817 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.960860 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:14:51.960932 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.958347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37055 (local address 127.0.98.3:45989)
0927 21:14:51.958399 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.958400 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.958510 (+ 110us) server_negotiation.cc:407] Connection header received
0927 21:14:51.958559 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.958560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.958573 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.958631 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:51.958778 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.959227 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.960201 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.960604 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.960672 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.960683 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.960816 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.960859 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:14:51.960964 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.958228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44295 (local address 127.0.98.1:41023)
0927 21:14:51.958300 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:51.958316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:51.958367 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:51.958517 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:51.958518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:51.958520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:51.958597 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.958599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.959290 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.959291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.960102 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.960104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.960672 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.960673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.960754 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.960757 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:14:51.960803 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:14:51.960852 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 21:14:51.960980 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.958234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45989 (local address 127.0.98.1:37055)
0927 21:14:51.958447 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:51.958457 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:51.958508 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:51.958640 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:51.958641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:51.958642 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:51.958713 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.958714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.959310 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.959311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.960105 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:51.960107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.960690 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.960690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:51.960751 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.960755 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:14:51.960801 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:14:51.960853 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":47,"thread_start_us":115,"threads_started":1}
I20260927 21:14:51.961337 14541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244231396ac14d1b99727bfacb136770" candidate_uuid: "14859606b1ac4cf280ba942361bf0fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47af539ed94c47eba5b6bd07687464b6" is_pre_election: true
I20260927 21:14:51.961501 14541 raft_consensus.cc:2454] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14859606b1ac4cf280ba942361bf0fa0 in term 0.
I20260927 21:14:51.961598 14407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244231396ac14d1b99727bfacb136770" candidate_uuid: "14859606b1ac4cf280ba942361bf0fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f74a8a0d484781807632e23bdea3eb" is_pre_election: true
I20260927 21:14:51.961738 14407 raft_consensus.cc:2454] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14859606b1ac4cf280ba942361bf0fa0 in term 0.
I20260927 21:14:51.961745 14207 leader_election.cc:302] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14859606b1ac4cf280ba942361bf0fa0, 47af539ed94c47eba5b6bd07687464b6; no voters:
I20260927 21:14:51.961871 14611 raft_consensus.cc:2781] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:14:51.961942 14611 raft_consensus.cc:482] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:14:51.961971 14611 raft_consensus.cc:3037] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:51.962803 14611 raft_consensus.cc:504] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:51.962934 14611 leader_election.cc:288] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 1 election: Requested vote from peers 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989), f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295)
I20260927 21:14:51.963092 14541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244231396ac14d1b99727bfacb136770" candidate_uuid: "14859606b1ac4cf280ba942361bf0fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47af539ed94c47eba5b6bd07687464b6"
I20260927 21:14:51.963106 14407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244231396ac14d1b99727bfacb136770" candidate_uuid: "14859606b1ac4cf280ba942361bf0fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f74a8a0d484781807632e23bdea3eb"
I20260927 21:14:51.963174 14541 raft_consensus.cc:3037] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:51.963182 14407 raft_consensus.cc:3037] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:51.963897 14541 raft_consensus.cc:2454] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14859606b1ac4cf280ba942361bf0fa0 in term 1.
I20260927 21:14:51.963930 14407 raft_consensus.cc:2454] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14859606b1ac4cf280ba942361bf0fa0 in term 1.
I20260927 21:14:51.964066 14207 leader_election.cc:302] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14859606b1ac4cf280ba942361bf0fa0, 47af539ed94c47eba5b6bd07687464b6; no voters:
I20260927 21:14:51.964186 14611 raft_consensus.cc:2781] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:14:51.964353 14611 raft_consensus.cc:686] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 LEADER]: Becoming Leader. State: Replica: 14859606b1ac4cf280ba942361bf0fa0, State: Running, Role: LEADER
I20260927 21:14:51.964464 14611 consensus_queue.cc:237] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:51.965339 14127 catalog_manager.cc:5495] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14859606b1ac4cf280ba942361bf0fa0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "14859606b1ac4cf280ba942361bf0fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } health_report { overall_health: UNKNOWN } } }
I20260927 21:14:51.978798 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.976330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.0.98.1:44677)
0927 21:14:51.976474 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:51.976477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:51.976498 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:14:51.976540 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:51.976541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:51.976563 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:51.976648 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:14:51.976792 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.977316 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.978074 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:51.978262 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:51.978319 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:51.978471 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:14:51.978582 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:14:51.978656 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:51.978676 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 21:14:51.978699 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:14:51.980909 14541 raft_consensus.cc:1261] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [term 1 FOLLOWER]: Refusing update from remote peer 14859606b1ac4cf280ba942361bf0fa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:14:51.980918 14407 raft_consensus.cc:1261] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [term 1 FOLLOWER]: Refusing update from remote peer 14859606b1ac4cf280ba942361bf0fa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:14:51.981256 14614 consensus_queue.cc:1035] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:14:51.981462 14611 consensus_queue.cc:1035] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:14:51.981849 14454 tablet.cc:2369] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:14:51.984006 14617 mvcc.cc:205] Tried to move back new op lower bound from 7334066962350637056 to 7334066962287136768. Current Snapshot: MvccSnapshot[applied={T|T < 7334066962350637056}]
I20260927 21:14:51.984200 14618 mvcc.cc:205] Tried to move back new op lower bound from 7334066962350637056 to 7334066962287136768. Current Snapshot: MvccSnapshot[applied={T|T < 7334066962350637056}]
I20260927 21:14:51.987025 14127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:14:51.987147 14127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:14:51.987912 14127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Sending DeleteTablet for 3 replicas of tablet 244231396ac14d1b99727bfacb136770
I20260927 21:14:51.988226 14253 tablet_service.cc:1514] Processing DeleteTablet for tablet 244231396ac14d1b99727bfacb136770 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35814
I20260927 21:14:51.988324 14387 tablet_service.cc:1514] Processing DeleteTablet for tablet 244231396ac14d1b99727bfacb136770 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50020
I20260927 21:14:51.988324 14521 tablet_service.cc:1514] Processing DeleteTablet for tablet 244231396ac14d1b99727bfacb136770 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51802
I20260927 21:14:51.988540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 14190
I20260927 21:14:51.989118 14627 tablet_replica.cc:330] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: stopping tablet replica
I20260927 21:14:51.989233 14627 raft_consensus.cc:2229] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:14:51.989362 14627 raft_consensus.cc:2258] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:14:51.989527 14628 tablet_replica.cc:330] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: stopping tablet replica
I20260927 21:14:51.989624 14628 raft_consensus.cc:2229] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:14:51.989713 14628 raft_consensus.cc:2258] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:14:51.990204 14627 ts_tablet_manager.cc:1905] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:14:51.990284 14628 ts_tablet_manager.cc:1905] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:14:51.992321 14627 ts_tablet_manager.cc:1918] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:14:51.992344 14628 ts_tablet_manager.cc:1918] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:14:51.992408 14627 log.cc:1196] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/244231396ac14d1b99727bfacb136770
I20260927 21:14:51.992413 14628 log.cc:1196] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/244231396ac14d1b99727bfacb136770
I20260927 21:14:51.992676 14627 ts_tablet_manager.cc:1939] T 244231396ac14d1b99727bfacb136770 P 47af539ed94c47eba5b6bd07687464b6: Deleting consensus metadata
I20260927 21:14:51.992678 14628 ts_tablet_manager.cc:1939] T 244231396ac14d1b99727bfacb136770 P f5f74a8a0d484781807632e23bdea3eb: Deleting consensus metadata
I20260927 21:14:51.993144 14112 catalog_manager.cc:4841] TS f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295): tablet 244231396ac14d1b99727bfacb136770 (table test-table [id=e54ec078f72740c6a0216563c30e8648]) successfully deleted
I20260927 21:14:51.993151 14113 catalog_manager.cc:4841] TS 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989): tablet 244231396ac14d1b99727bfacb136770 (table test-table [id=e54ec078f72740c6a0216563c30e8648]) successfully deleted
W20260927 21:14:51.994382 14115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44677 (error 108)
I20260927 21:14:51.994961 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:51.994820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44677 (local address 127.0.0.1:35820)
0927 21:14:51.994882 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:51.994925 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:14:51.994961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 14324
W20260927 21:14:51.997725 14115 catalog_manager.cc:4584] TS 14859606b1ac4cf280ba942361bf0fa0 (127.0.98.1:44677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 244231396ac14d1b99727bfacb136770: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44677: connect: Connection refused (error 111)
I20260927 21:14:51.999534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 14458
I20260927 21:14:52.004546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 14096
I20260927 21:14:52.009588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33899
--webserver_interface=127.0.98.62
--webserver_port=36317
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:52.095863 14630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:52.096166 14630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:52.096210 14630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:52.098270 14630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:14:52.098337 14630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:52.098356 14630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:52.098374 14630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:14:52.098390 14630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:14:52.100301 14630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:52.100605 14630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:52.100835 14630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:52.110071 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:52.110082 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:52.110082 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:52.110670 14630 server_base.cc:1034] running on GCE node
I20260927 21:14:52.111024 14630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:52.111393 14630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:52.112572 14630 hybrid_clock.cc:648] HybridClock initialized: now 1790543692112557 us; error 38 us; skew 500 ppm
Sep 27 21:14:52 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:52.115414 14630 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:14:52.115922 14630 webserver.cc:460] Webserver started at http://127.0.98.62:36317/ using document root <none> and password file <none>
I20260927 21:14:52.116158 14630 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:52.116225 14630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:52.117722 14630 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:52.118381 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:52.118542 14630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:52.118631 14630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "57984bfd275a48d48bda0f10d8121810"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "cd31874de69eb6871c3bd14663de5518"
server_key_iv: "01990b88c71d5530da2849d31246af6f"
server_key_version: "encryptionkey@0"
I20260927 21:14:52.119076 14630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:52.133574 14630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:52.135622 14630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:52.135774 14630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:52.139755 14630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33899
I20260927 21:14:52.139822 14698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33899 every 8 connection(s)
I20260927 21:14:52.140168 14630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:14:52.141827 14699 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:14:52.142445 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Bootstrap starting.
I20260927 21:14:52.144681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 14630
I20260927 21:14:52.144956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44677
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33899
--webserver_port=36363
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:14:52.150933 14699 log.cc:824] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:52.153110 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:14:52.153430 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Bootstrap complete.
I20260927 21:14:52.155686 14699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:52.155939 14699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57984bfd275a48d48bda0f10d8121810, State: Initialized, Role: FOLLOWER
I20260927 21:14:52.156104 14699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:52.156172 14699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:14:52.156220 14699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:14:52.156268 14699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:14:52.157438 14699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:52.157549 14699 leader_election.cc:302] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57984bfd275a48d48bda0f10d8121810; no voters:
I20260927 21:14:52.157725 14699 leader_election.cc:288] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:14:52.157776 14703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:14:52.157941 14703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 2 LEADER]: Becoming Leader. State: Replica: 57984bfd275a48d48bda0f10d8121810, State: Running, Role: LEADER
I20260927 21:14:52.158032 14699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:14:52.158062 14703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:52.158432 14703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57984bfd275a48d48bda0f10d8121810. Latest consensus state: current_term: 2 leader_uuid: "57984bfd275a48d48bda0f10d8121810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } } }
I20260927 21:14:52.158712 14703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:52.158638 14704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57984bfd275a48d48bda0f10d8121810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } } }
I20260927 21:14:52.159094 14712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:14:52.159113 14704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:52.160027 14712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:14:52.160203 14712 catalog_manager.cc:1259] Loaded cluster ID: f06d2409478b49b39ef35ef4cd38d36d
I20260927 21:14:52.160418 14712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:14:52.160707 14712 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 21:14:52.165697 14712 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 21:14:52.168395 14712 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:14:52.168628 14712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Loaded TSK: 0
I20260927 21:14:52.168840 14712 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:14:52.234781 14701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:52.235008 14701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:52.235033 14701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:52.236312 14701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:52.236369 14701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:52.236399 14701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:14:52.238026 14701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36363
--tserver_master_addrs=127.0.98.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:52.238303 14701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:52.238528 14701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:52.243619 14729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:52.243615 14726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:52.243609 14727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:52.243813 14701 server_base.cc:1034] running on GCE node
I20260927 21:14:52.244099 14701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:52.244354 14701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:52.245549 14701 hybrid_clock.cc:648] HybridClock initialized: now 1790543692245523 us; error 37 us; skew 500 ppm
Sep 27 21:14:52 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:52.248283 14701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:14:52.248759 14701 webserver.cc:460] Webserver started at http://127.0.98.1:36363/ using document root <none> and password file <none>
I20260927 21:14:52.248978 14701 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:52.249045 14701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:52.251125 14701 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:52.252106 14736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:52.252343 14701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:52.252450 14701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "14859606b1ac4cf280ba942361bf0fa0"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "2dffac691a3cb07616307d47b3b60fdb"
server_key_iv: "7f030ad0ce8c726e879414ffc9e0fcbf"
server_key_version: "encryptionkey@0"
I20260927 21:14:52.252825 14701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:52.278497 14701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:52.281443 14701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:52.281600 14701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:52.281831 14701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:52.282261 14743 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:14:52.284227 14701 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:14:52.284309 14701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:14:52.284348 14701 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:14:52.285022 14701 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:14:52.285058 14701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:14:52.285142 14743 tablet_bootstrap.cc:492] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Bootstrap starting.
I20260927 21:14:52.291828 14701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44677
I20260927 21:14:52.291944 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44677 every 8 connection(s)
I20260927 21:14:52.292320 14701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:14:52.295831 14743 log.cc:824] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:52.298421 14743 tablet_bootstrap.cc:492] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:14:52.298700 14743 tablet_bootstrap.cc:492] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Bootstrap complete.
Sep 27 21:14:52 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:52.299711 14743 ts_tablet_manager.cc:1397] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 21:14:52.300582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 14701
I20260927 21:14:52.300858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44295
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33899
--webserver_port=37475
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:14:52.300880 14743 raft_consensus.cc:348] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:52.301223 14743 raft_consensus.cc:729] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14859606b1ac4cf280ba942361bf0fa0, State: Initialized, Role: FOLLOWER
I20260927 21:14:52.301422 14743 consensus_queue.cc:260] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } }
I20260927 21:14:52.301715 14743 ts_tablet_manager.cc:1428] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260927 21:14:52.303777 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.293090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39405 (local address 127.0.98.62:33899)
0927 21:14:52.293469 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.293477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.293515 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:14:52.294145 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.294150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.294193 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.294310 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:52.294942 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.295802 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.296961 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.297419 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.297519 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.300041 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:52.300045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:52.300045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:52.300056 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:52.302056 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.302583 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.302586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.302586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.302674 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.302922 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.302922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.302923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.303140 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:52.303202 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.303513 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.303621 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":75,"threads_started":1}
I20260927 21:14:52.303861 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.293011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:39405)
0927 21:14:52.293375 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:52.293427 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:52.293924 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:52.294351 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:52.294355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:52.294635 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:52.294826 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.294832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.295922 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.295923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.296870 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.296874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.297522 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.297523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.297651 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.298259 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:52.298276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:52.299915 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:52.302172 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:52.302180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:52.302183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:52.302498 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:52.302733 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:52.302734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:52.302734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:52.302863 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:52.303203 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:52.303209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:52.303365 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 21:14:52.303475 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:14:52.303651 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":89,"threads_started":1}
I20260927 21:14:52.304566 14851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33899
I20260927 21:14:52.304683 14851 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:52.304872 14851 heartbeater.cc:511] Master 127.0.98.62:33899 requested a full tablet report, sending...
I20260927 21:14:52.305691 14663 ts_manager.cc:194] Registered new tserver with Master: 14859606b1ac4cf280ba942361bf0fa0 (127.0.98.1:44677)
I20260927 21:14:52.306984 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39405
I20260927 21:14:52.308207 14851 heartbeater.cc:503] Master 127.0.98.62:33899 was elected leader, sending a full tablet report...
Sep 27 21:14:52 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:14:52.314071 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.306432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.0.98.1:44677)
0927 21:14:52.306578 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.306583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.306728 (+ 145us) server_negotiation.cc:407] Connection header received
0927 21:14:52.306804 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.306806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.306823 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.306884 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:52.307310 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.308021 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.308925 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.309202 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.309258 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.311473 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:52.311480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:52.311481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:52.311487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:52.312843 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.313215 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.313218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.313218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.313299 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.313476 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.313477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.313478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.313626 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:52.313699 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.313871 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.313977 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260927 21:14:52.314071 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.306365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44677 (local address 127.0.0.1:35824)
0927 21:14:52.306636 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:52.306662 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:52.306731 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:52.306892 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:52.306896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:52.307171 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:52.307221 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.307224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.308095 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.308096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.308851 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.308855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.309285 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.309290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.309374 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.309879 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:52.309883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:52.311373 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:52.312928 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:52.312936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:52.312938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:52.313140 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:52.313356 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:52.313357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:52.313357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:52.313426 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:52.313685 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:52.313692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:52.313749 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:14:52.313825 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:14:52.313961 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":115,"threads_started":1}
I20260927 21:14:52.314473 14785 tablet_service.cc:1514] Processing DeleteTablet for tablet 244231396ac14d1b99727bfacb136770 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35824
I20260927 21:14:52.315002 14860 tablet_replica.cc:330] stopping tablet replica
I20260927 21:14:52.315121 14860 raft_consensus.cc:2229] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:14:52.315191 14860 raft_consensus.cc:2258] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:14:52.315505 14860 ts_tablet_manager.cc:1905] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:14:52.318008 14860 ts_tablet_manager.cc:1918] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:14:52.318099 14860 log.cc:1196] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/244231396ac14d1b99727bfacb136770
I20260927 21:14:52.318310 14860 ts_tablet_manager.cc:1939] T 244231396ac14d1b99727bfacb136770 P 14859606b1ac4cf280ba942361bf0fa0: Deleting consensus metadata
I20260927 21:14:52.318964 14650 catalog_manager.cc:4841] TS 14859606b1ac4cf280ba942361bf0fa0 (127.0.98.1:44677): tablet 244231396ac14d1b99727bfacb136770 (table test-table [id=e54ec078f72740c6a0216563c30e8648]) successfully deleted
W20260927 21:14:52.386682 14857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:52.386910 14857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:52.386936 14857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:52.388242 14857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:52.388302 14857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:52.388332 14857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:14:52.390614 14857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37475
--tserver_master_addrs=127.0.98.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:52.390908 14857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:52.391239 14857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:52.396296 14868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:52.396405 14865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:52.396315 14866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:52.396427 14857 server_base.cc:1034] running on GCE node
I20260927 21:14:52.396719 14857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:52.396955 14857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:52.398133 14857 hybrid_clock.cc:648] HybridClock initialized: now 1790543692398117 us; error 34 us; skew 500 ppm
Sep 27 21:14:52 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:52.400719 14857 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:14:52.401161 14857 webserver.cc:460] Webserver started at http://127.0.98.2:37475/ using document root <none> and password file <none>
I20260927 21:14:52.401392 14857 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:52.401453 14857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:52.402781 14857 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:52.403378 14875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:52.403537 14857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:14:52.403618 14857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "f5f74a8a0d484781807632e23bdea3eb"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "07b019ac1b512126c8f5f081f1ad1584"
server_key_iv: "2435e2161c306ff0aff590ed53fbf963"
server_key_version: "encryptionkey@0"
I20260927 21:14:52.403981 14857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:52.411584 14857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:52.413694 14857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:52.413841 14857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:52.414067 14857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:52.414348 14857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:52.414381 14857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:52.414409 14857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:52.414433 14857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:52.419806 14857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44295
I20260927 21:14:52.419853 14988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44295 every 8 connection(s)
I20260927 21:14:52.420181 14857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:14:52 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:52.426661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 14857
I20260927 21:14:52.426973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45989
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33899
--webserver_port=35475
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:14:52.429792 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.420875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48935 (local address 127.0.98.62:33899)
0927 21:14:52.420930 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.420932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.421124 (+ 192us) server_negotiation.cc:407] Connection header received
0927 21:14:52.421676 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.421677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.421687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.421752 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:52.422687 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.423218 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.424239 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.424459 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.424525 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.426877 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:52.426879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:52.426879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:52.426886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:52.428211 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.428722 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.428723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.428723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.428769 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.429051 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.429052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.429052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.429192 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:52.429257 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.429644 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.429724 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:14:52.430004 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.420813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:48935)
0927 21:14:52.420972 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:52.421014 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:52.421535 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:52.421814 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:52.421818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:52.422416 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:52.422604 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.422608 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.423295 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.423296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.424134 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.424136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.424563 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.424564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.424681 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.425188 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:52.425197 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:52.426771 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:52.428297 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:52.428308 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:52.428311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:52.428648 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:52.428822 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:52.428823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:52.428823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:52.428962 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:52.429285 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:52.429291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:52.429485 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 21:14:52.429606 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 21:14:52.429777 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260927 21:14:52.430487 14989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33899
I20260927 21:14:52.430588 14989 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:52.430766 14989 heartbeater.cc:511] Master 127.0.98.62:33899 requested a full tablet report, sending...
I20260927 21:14:52.431164 14663 ts_manager.cc:194] Registered new tserver with Master: f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295)
I20260927 21:14:52.431684 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48935
W20260927 21:14:52.509142 14992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:52.509398 14992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:52.509426 14992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:52.510632 14992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:52.510672 14992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:52.510699 14992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:14:52.512162 14992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35475
--tserver_master_addrs=127.0.98.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:52.512398 14992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:52.512599 14992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:52.517629 14998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:52.517648 14997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:52.517629 15000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:52.517704 14992 server_base.cc:1034] running on GCE node
I20260927 21:14:52.518023 14992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:52.518292 14992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:52.519454 14992 hybrid_clock.cc:648] HybridClock initialized: now 1790543692519433 us; error 33 us; skew 500 ppm
Sep 27 21:14:52 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:52.522830 14992 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:14:52.523401 14992 webserver.cc:460] Webserver started at http://127.0.98.3:35475/ using document root <none> and password file <none>
I20260927 21:14:52.523653 14992 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:52.523727 14992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:52.525630 14992 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:14:52.526289 15007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:52.526430 14992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:14:52.526513 14992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "47af539ed94c47eba5b6bd07687464b6"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "c787cff65469c395d19c57ef294b4237"
server_key_iv: "cc33550399d9297903ece970f18623e6"
server_key_version: "encryptionkey@0"
I20260927 21:14:52.526863 14992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:52.541994 14992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:52.543960 14992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:52.544102 14992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:52.544317 14992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:52.544602 14992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:52.544636 14992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:52.544663 14992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:52.544680 14992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:52.550271 14992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45989
I20260927 21:14:52.550324 15120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45989 every 8 connection(s)
I20260927 21:14:52.550663 14992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:14:52.553512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 14992
Sep 27 21:14:52 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:14:52.560137 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.553870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.0.98.62:33899)
0927 21:14:52.553988 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.553989 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.554040 (+ 51us) server_negotiation.cc:407] Connection header received
0927 21:14:52.554103 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.554104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.554114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.554170 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:52.554701 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.555384 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.556132 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.556338 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.556389 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.557585 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:52.557586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:52.557587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:52.557593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:52.559038 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.559418 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.559420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.559420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.559475 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.559635 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.559636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.559636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.559762 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:52.559913 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.559959 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.560038 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":50,"threads_started":1}
I20260927 21:14:52.560261 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.551365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53361 (local address 127.0.98.62:33899)
0927 21:14:52.551413 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.551414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.551625 (+ 211us) server_negotiation.cc:407] Connection header received
0927 21:14:52.552142 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.552144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.552156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.552211 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:52.553210 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.553818 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.554820 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.555124 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.555210 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.557399 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:52.557401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:52.557401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:52.557408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:52.558911 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.559351 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.559352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.559352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.559409 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:52.559640 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:52.559640 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:52.559641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:52.559772 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:52.559826 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.560100 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.560178 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:14:52.560467 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.551351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:53361)
0927 21:14:52.551501 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:52.551537 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:52.552037 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:52.552295 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:52.552299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:52.552895 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:52.553100 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.553105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.553936 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.553938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.554746 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.554748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.555227 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.555228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.555339 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.555838 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:52.555847 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:52.557301 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:52.558987 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:52.558995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:52.558997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:52.559293 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:52.559473 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:52.559473 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:52.559474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:52.559583 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:52.559828 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:52.559834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:52.559980 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 21:14:52.560072 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:14:52.560234 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260927 21:14:52.560848 15121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33899
I20260927 21:14:52.560946 15121 heartbeater.cc:464] Registering TS with master...
I20260927 21:14:52.561132 15121 heartbeater.cc:511] Master 127.0.98.62:33899 requested a full tablet report, sending...
I20260927 21:14:52.561492 14663 ts_manager.cc:194] Registered new tserver with Master: 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989)
I20260927 21:14:52.561851 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53361
I20260927 21:14:52.572129 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:14:52.574992 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.572498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.0.98.62:33899)
0927 21:14:52.572567 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.572569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.572651 (+ 82us) server_negotiation.cc:407] Connection header received
0927 21:14:52.572735 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.572735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.572791 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.572855 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:14:52.572983 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.573531 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.574308 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.574493 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.574548 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.574695 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:14:52.574801 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:14:52.574874 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.574897 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.574917 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:14:52.575289 14663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:14:52.576814 14663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:14:52.578949 14785 tablet_service.cc:1467] Processing CreateTablet for tablet 48e147d86d8c44fda29f17f6595815d5 (DEFAULT_TABLE table=test-table [id=fdbc3239ae854d2581bcc75e0f721514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:14:52.579166 14785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e147d86d8c44fda29f17f6595815d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:52.580776 14743 tablet_bootstrap.cc:492] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Bootstrap starting.
I20260927 21:14:52.581139 14743 tablet_bootstrap.cc:654] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:52.582193 14743 tablet_bootstrap.cc:492] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: No bootstrap required, opened a new log
I20260927 21:14:52.582264 14743 ts_tablet_manager.cc:1397] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:14:52.582427 14743 raft_consensus.cc:348] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:52.582506 14743 raft_consensus.cc:374] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:52.582564 14743 raft_consensus.cc:729] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14859606b1ac4cf280ba942361bf0fa0, State: Initialized, Role: FOLLOWER
I20260927 21:14:52.582621 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.578638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44295 (local address 127.0.0.1:50026)
0927 21:14:52.578671 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:52.578681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:52.578715 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:52.579224 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:52.579225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:52.579227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:52.579284 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.579286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.580382 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.580383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.581474 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.581489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.582364 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.582365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.582439 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.582458 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 21:14:52.582492 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 21:14:52.582548 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 21:14:52.582614 14743 consensus_queue.cc:260] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:52.582715 14743 ts_tablet_manager.cc:1428] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:14:52.582901 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.578647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45989 (local address 127.0.0.1:51818)
0927 21:14:52.578845 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:52.578854 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:52.578898 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:52.579519 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:52.579520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:52.579522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:52.579559 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.579560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.580872 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.580873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.581744 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.581747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.582681 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.582682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.582746 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.582750 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:14:52.582797 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:14:52.582837 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":32,"thread_start_us":107,"threads_started":1}
I20260927 21:14:52.583033 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.578823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.0.98.2:44295)
0927 21:14:52.579038 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.579042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.579053 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:14:52.579095 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.579097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.579112 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.579206 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:52.579363 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.580283 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.581594 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.582263 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.582806 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.582878 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.582905 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.582952 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
I20260927 21:14:52.583129 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.578808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.0.98.3:45989)
0927 21:14:52.579370 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.579375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.579385 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:14:52.579420 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.579422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.579436 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.579514 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:52.579648 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.580669 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.581806 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.582457 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.582536 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.582603 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.583005 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.583052 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":84,"threads_started":1}
I20260927 21:14:52.583765 15055 tablet_service.cc:1467] Processing CreateTablet for tablet 48e147d86d8c44fda29f17f6595815d5 (DEFAULT_TABLE table=test-table [id=fdbc3239ae854d2581bcc75e0f721514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:14:52.584132 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e147d86d8c44fda29f17f6595815d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:52.585690 14923 tablet_service.cc:1467] Processing CreateTablet for tablet 48e147d86d8c44fda29f17f6595815d5 (DEFAULT_TABLE table=test-table [id=fdbc3239ae854d2581bcc75e0f721514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:14:52.585968 14923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e147d86d8c44fda29f17f6595815d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:14:52.587014 15131 tablet_bootstrap.cc:492] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: Bootstrap starting.
I20260927 21:14:52.587572 15131 tablet_bootstrap.cc:654] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:52.587993 15131 log.cc:824] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:52.588650 15131 tablet_bootstrap.cc:492] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: No bootstrap required, opened a new log
I20260927 21:14:52.588660 15133 tablet_bootstrap.cc:492] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: Bootstrap starting.
I20260927 21:14:52.588721 15131 ts_tablet_manager.cc:1397] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:14:52.589334 15133 tablet_bootstrap.cc:654] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: Neither blocks nor log segments found. Creating new log.
I20260927 21:14:52.589872 15133 log.cc:824] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:52.590713 15133 tablet_bootstrap.cc:492] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: No bootstrap required, opened a new log
I20260927 21:14:52.590620 15131 raft_consensus.cc:348] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:52.590782 15133 ts_tablet_manager.cc:1397] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:14:52.590790 15131 raft_consensus.cc:374] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:52.590823 15131 raft_consensus.cc:729] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47af539ed94c47eba5b6bd07687464b6, State: Initialized, Role: FOLLOWER
I20260927 21:14:52.590939 15131 consensus_queue.cc:260] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:52.591187 15131 ts_tablet_manager.cc:1428] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:14:52.591225 15121 heartbeater.cc:503] Master 127.0.98.62:33899 was elected leader, sending a full tablet report...
I20260927 21:14:52.592463 15133 raft_consensus.cc:348] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:52.592607 15133 raft_consensus.cc:374] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:14:52.592638 15133 raft_consensus.cc:729] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f74a8a0d484781807632e23bdea3eb, State: Initialized, Role: FOLLOWER
I20260927 21:14:52.592751 15133 consensus_queue.cc:260] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:52.593021 15133 ts_tablet_manager.cc:1428] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:14:52.593044 14989 heartbeater.cc:503] Master 127.0.98.62:33899 was elected leader, sending a full tablet report...
W20260927 21:14:52.670753 14990 tablet.cc:2369] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:14:52.793762 14852 tablet.cc:2369] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:14:52.801254 15122 tablet.cc:2369] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:14:52.958050 14856 raft_consensus.cc:482] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:14:52.958185 14856 raft_consensus.cc:504] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:52.958505 14856 leader_election.cc:288] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295), 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989)
I20260927 21:14:52.961587 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.958787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58613 (local address 127.0.98.3:45989)
0927 21:14:52.958830 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.958833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.958906 (+ 73us) server_negotiation.cc:407] Connection header received
0927 21:14:52.959009 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.959010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.959021 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.959154 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:52.959348 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.959803 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.960823 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.961223 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.961335 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.961346 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.961470 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.961513 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:14:52.961597 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.958768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49595 (local address 127.0.98.2:44295)
0927 21:14:52.958829 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:52.958832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:52.958907 (+ 75us) server_negotiation.cc:407] Connection header received
0927 21:14:52.959014 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:52.959015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:52.959027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:52.959108 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:14:52.959298 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.959789 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.960843 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.961200 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.961251 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.961260 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:52.961488 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 21:14:52.961529 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:14:52.961633 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.958691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45989 (local address 127.0.98.1:58613)
0927 21:14:52.958838 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:52.958850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:52.958938 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:52.959195 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:52.959196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:52.959199 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:52.959267 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.959269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.959887 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.959888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.960719 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.960721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.961339 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.961340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.961406 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.961410 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:14:52.961456 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:14:52.961508 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":28,"thread_start_us":104,"threads_started":1}
I20260927 21:14:52.961645 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:52.958701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44295 (local address 127.0.98.1:49595)
0927 21:14:52.958855 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:52.958863 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:52.958938 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:52.959140 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:52.959140 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:52.959143 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:14:52.959206 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.959208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.959881 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.959882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.960736 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:52.960739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:52.961298 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:52.961299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:52.961378 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:52.961406 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 21:14:52.961463 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 21:14:52.961509 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":96,"thread_start_us":58,"threads_started":1}
I20260927 21:14:52.962142 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e147d86d8c44fda29f17f6595815d5" candidate_uuid: "14859606b1ac4cf280ba942361bf0fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47af539ed94c47eba5b6bd07687464b6" is_pre_election: true
I20260927 21:14:52.962142 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e147d86d8c44fda29f17f6595815d5" candidate_uuid: "14859606b1ac4cf280ba942361bf0fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f74a8a0d484781807632e23bdea3eb" is_pre_election: true
I20260927 21:14:52.962314 15075 raft_consensus.cc:2454] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14859606b1ac4cf280ba942361bf0fa0 in term 0.
I20260927 21:14:52.962314 14943 raft_consensus.cc:2454] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14859606b1ac4cf280ba942361bf0fa0 in term 0.
I20260927 21:14:52.962541 14737 leader_election.cc:302] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14859606b1ac4cf280ba942361bf0fa0, f5f74a8a0d484781807632e23bdea3eb; no voters:
I20260927 21:14:52.962684 14856 raft_consensus.cc:2781] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:14:52.962744 14856 raft_consensus.cc:482] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:14:52.962760 14856 raft_consensus.cc:3037] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:52.963405 14856 raft_consensus.cc:504] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:52.963548 14856 leader_election.cc:288] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 1 election: Requested vote from peers f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295), 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989)
I20260927 21:14:52.963752 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e147d86d8c44fda29f17f6595815d5" candidate_uuid: "14859606b1ac4cf280ba942361bf0fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f74a8a0d484781807632e23bdea3eb"
I20260927 21:14:52.963752 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e147d86d8c44fda29f17f6595815d5" candidate_uuid: "14859606b1ac4cf280ba942361bf0fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47af539ed94c47eba5b6bd07687464b6"
I20260927 21:14:52.963845 14943 raft_consensus.cc:3037] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:52.963850 15075 raft_consensus.cc:3037] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:14:52.964732 14943 raft_consensus.cc:2454] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14859606b1ac4cf280ba942361bf0fa0 in term 1.
I20260927 21:14:52.964732 15075 raft_consensus.cc:2454] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14859606b1ac4cf280ba942361bf0fa0 in term 1.
I20260927 21:14:52.964901 14738 leader_election.cc:302] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14859606b1ac4cf280ba942361bf0fa0, 47af539ed94c47eba5b6bd07687464b6; no voters:
I20260927 21:14:52.964988 14856 raft_consensus.cc:2781] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:14:52.965122 14856 raft_consensus.cc:686] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 LEADER]: Becoming Leader. State: Replica: 14859606b1ac4cf280ba942361bf0fa0, State: Running, Role: LEADER
I20260927 21:14:52.965190 14856 consensus_queue.cc:237] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:52.965947 14663 catalog_manager.cc:5495] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14859606b1ac4cf280ba942361bf0fa0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "14859606b1ac4cf280ba942361bf0fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } health_report { overall_health: HEALTHY } } }
I20260927 21:14:53.023289 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.020746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.0.98.1:44677)
0927 21:14:53.020908 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:53.020909 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:53.020916 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:14:53.020959 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:53.020960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:53.020979 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:53.021042 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:14:53.021178 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:53.021709 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:53.022501 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:53.022712 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:53.022783 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:53.022880 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:14:53.023011 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:14:53.023087 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:53.023152 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 21:14:53.023176 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260927 21:14:53.024940 14943 raft_consensus.cc:1261] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [term 1 FOLLOWER]: Refusing update from remote peer 14859606b1ac4cf280ba942361bf0fa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:14:53.024941 15075 raft_consensus.cc:1261] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [term 1 FOLLOWER]: Refusing update from remote peer 14859606b1ac4cf280ba942361bf0fa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:14:53.025295 15139 consensus_queue.cc:1035] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:14:53.025435 14856 consensus_queue.cc:1035] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:14:53.028965 15144 mvcc.cc:205] Tried to move back new op lower bound from 7334066966627459072 to 7334066966386094080. Current Snapshot: MvccSnapshot[applied={T|T < 7334066966627459072 or (T in {7334066966627459072})}]
I20260927 21:14:53.030151 14662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:14:53.030260 14662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:14:53.030977 14662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Sending DeleteTablet for 3 replicas of tablet 48e147d86d8c44fda29f17f6595815d5
I20260927 21:14:53.031270 14923 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e147d86d8c44fda29f17f6595815d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50026
I20260927 21:14:53.031294 14785 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e147d86d8c44fda29f17f6595815d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35824
I20260927 21:14:53.031383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 14701
I20260927 21:14:53.031487 15055 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e147d86d8c44fda29f17f6595815d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:14:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51818
I20260927 21:14:53.031824 15151 tablet_replica.cc:330] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: stopping tablet replica
I20260927 21:14:53.031999 15151 raft_consensus.cc:2229] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:14:53.032119 15151 raft_consensus.cc:2258] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:14:53.032493 15151 ts_tablet_manager.cc:1905] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:14:53.033702 15153 tablet_replica.cc:330] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: stopping tablet replica
I20260927 21:14:53.033813 15153 raft_consensus.cc:2229] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:14:53.033924 15153 raft_consensus.cc:2258] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:14:53.033962 15151 ts_tablet_manager.cc:1918] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:14:53.034039 15151 log.cc:1196] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/48e147d86d8c44fda29f17f6595815d5
I20260927 21:14:53.034384 15153 ts_tablet_manager.cc:1905] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:14:53.034415 15151 ts_tablet_manager.cc:1939] T 48e147d86d8c44fda29f17f6595815d5 P f5f74a8a0d484781807632e23bdea3eb: Deleting consensus metadata
I20260927 21:14:53.034837 14647 catalog_manager.cc:4841] TS f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295): tablet 48e147d86d8c44fda29f17f6595815d5 (table test-table [id=fdbc3239ae854d2581bcc75e0f721514]) successfully deleted
I20260927 21:14:53.035828 15153 ts_tablet_manager.cc:1918] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:14:53.035887 15153 log.cc:1196] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/48e147d86d8c44fda29f17f6595815d5
I20260927 21:14:53.036108 15153 ts_tablet_manager.cc:1939] T 48e147d86d8c44fda29f17f6595815d5 P 47af539ed94c47eba5b6bd07687464b6: Deleting consensus metadata
I20260927 21:14:53.036504 14648 catalog_manager.cc:4841] TS 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989): tablet 48e147d86d8c44fda29f17f6595815d5 (table test-table [id=fdbc3239ae854d2581bcc75e0f721514]) successfully deleted
W20260927 21:14:53.037009 14650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44677 (error 108)
I20260927 21:14:53.037179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 14857
I20260927 21:14:53.038436 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.038309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44677 (local address 127.0.0.1:35838)
0927 21:14:53.038367 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.038408 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 21:14:53.038559 14650 catalog_manager.cc:4584] TS 14859606b1ac4cf280ba942361bf0fa0 (127.0.98.1:44677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48e147d86d8c44fda29f17f6595815d5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44677: connect: Connection refused (error 111)
I20260927 21:14:53.041738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 14992
I20260927 21:14:53.046697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 14630
I20260927 21:14:53.051366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33899
--webserver_interface=127.0.98.62
--webserver_port=36317
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:53.132699 15155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:53.132937 15155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:53.132961 15155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:53.134274 15155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:14:53.134331 15155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:53.134346 15155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:53.134356 15155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:14:53.134367 15155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:14:53.135993 15155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:53.136241 15155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:53.136484 15155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:53.141392 15161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:53.141393 15160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:53.141573 15155 server_base.cc:1034] running on GCE node
W20260927 21:14:53.141392 15163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:53.141922 15155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:53.142223 15155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:53.143397 15155 hybrid_clock.cc:648] HybridClock initialized: now 1790543693143383 us; error 30 us; skew 500 ppm
Sep 27 21:14:53 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:53.145915 15155 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:14:53.146347 15155 webserver.cc:460] Webserver started at http://127.0.98.62:36317/ using document root <none> and password file <none>
I20260927 21:14:53.146575 15155 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:53.146620 15155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:53.147959 15155 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:53.148614 15170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:53.148763 15155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:53.148850 15155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "57984bfd275a48d48bda0f10d8121810"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "cd31874de69eb6871c3bd14663de5518"
server_key_iv: "01990b88c71d5530da2849d31246af6f"
server_key_version: "encryptionkey@0"
I20260927 21:14:53.149199 15155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:53.189587 15155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:53.191605 15155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:53.191722 15155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:53.195533 15155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33899
I20260927 21:14:53.195568 15222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33899 every 8 connection(s)
I20260927 21:14:53.195931 15155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:14:53.196431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 15155
I20260927 21:14:53.196667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44677
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33899
--webserver_port=36363
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:14:53.197620 15223 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:14:53.198256 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Bootstrap starting.
I20260927 21:14:53.206252 15223 log.cc:824] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:53.208391 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:14:53.208673 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810: Bootstrap complete.
I20260927 21:14:53.210789 15223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:53.211023 15223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57984bfd275a48d48bda0f10d8121810, State: Initialized, Role: FOLLOWER
I20260927 21:14:53.211175 15223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:53.211249 15223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:14:53.211299 15223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:14:53.211346 15223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:14:53.212353 15223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:53.212457 15223 leader_election.cc:302] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57984bfd275a48d48bda0f10d8121810; no voters:
I20260927 21:14:53.212639 15223 leader_election.cc:288] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:14:53.212688 15227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:14:53.212808 15227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [term 3 LEADER]: Becoming Leader. State: Replica: 57984bfd275a48d48bda0f10d8121810, State: Running, Role: LEADER
I20260927 21:14:53.212895 15227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } }
I20260927 21:14:53.212922 15223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:14:53.213241 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57984bfd275a48d48bda0f10d8121810. Latest consensus state: current_term: 3 leader_uuid: "57984bfd275a48d48bda0f10d8121810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } } }
I20260927 21:14:53.213256 15228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "57984bfd275a48d48bda0f10d8121810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57984bfd275a48d48bda0f10d8121810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33899 } } }
I20260927 21:14:53.213372 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:53.213379 15228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57984bfd275a48d48bda0f10d8121810 [sys.catalog]: This master's current role is: LEADER
I20260927 21:14:53.213579 15233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:14:53.214401 15233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:14:53.214499 15233 catalog_manager.cc:1259] Loaded cluster ID: f06d2409478b49b39ef35ef4cd38d36d
I20260927 21:14:53.214778 15233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 21:14:53.215068 15233 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7660a349ad google::LogMessage::Fail() at ??:0
@ 0x7f7660a3788a google::LogMessage::SendToLog() at ??:0
@ 0x7f7660a344de google::LogMessage::Flush() at ??:0
@ 0x7f7660a35a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f76689550f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f766895e8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f76629020d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f76628f7954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f766375b6db start_thread at ??:0
@ 0x7f765f53261f clone at ??:0
W20260927 21:14:53.289705 15225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:53.289983 15225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:53.290025 15225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:53.291428 15225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:53.291491 15225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:53.291532 15225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:14:53.293336 15225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36363
--tserver_master_addrs=127.0.98.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:53.293610 15225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:53.293864 15225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:53.299677 15259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:53.299774 15262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:53.299659 15260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:53.300175 15225 server_base.cc:1034] running on GCE node
I20260927 21:14:53.300387 15225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:53.300661 15225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:53.301877 15225 hybrid_clock.cc:648] HybridClock initialized: now 1790543693301851 us; error 40 us; skew 500 ppm
Sep 27 21:14:53 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:53.304492 15225 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:14:53.304996 15225 webserver.cc:460] Webserver started at http://127.0.98.1:36363/ using document root <none> and password file <none>
I20260927 21:14:53.305243 15225 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:53.305327 15225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:53.306773 15225 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:53.307523 15269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:53.307703 15225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:53.307777 15225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "14859606b1ac4cf280ba942361bf0fa0"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "2dffac691a3cb07616307d47b3b60fdb"
server_key_iv: "7f030ad0ce8c726e879414ffc9e0fcbf"
server_key_version: "encryptionkey@0"
I20260927 21:14:53.308171 15225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:53.325122 15225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:53.327971 15225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:53.328153 15225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:53.328393 15225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:53.328851 15276 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:14:53.330909 15225 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:14:53.330974 15225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:14:53.331012 15225 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:14:53.331679 15225 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:14:53.331713 15225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:14:53.331797 15276 tablet_bootstrap.cc:492] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Bootstrap starting.
I20260927 21:14:53.338394 15225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44677
I20260927 21:14:53.338487 15383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44677 every 8 connection(s)
I20260927 21:14:53.338814 15225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:14:53.341725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 15225
I20260927 21:14:53.341976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44295
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33899
--webserver_port=37475
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:14:53.342597 15276 log.cc:824] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:14:53.345448 15276 tablet_bootstrap.cc:492] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:14:53.345758 15276 tablet_bootstrap.cc:492] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Bootstrap complete.
I20260927 21:14:53.346838 15276 ts_tablet_manager.cc:1397] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.003s
I20260927 21:14:53.348361 15276 raft_consensus.cc:348] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:53.348722 15276 raft_consensus.cc:729] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14859606b1ac4cf280ba942361bf0fa0, State: Initialized, Role: FOLLOWER
I20260927 21:14:53.348863 15276 consensus_queue.cc:260] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:53.349166 15276 ts_tablet_manager.cc:1428] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260927 21:14:53.423151 15386 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:14:53.339471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:37983)
0927 21:14:53.339643 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.339687 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:53.340142 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:53.422966 (+ 82824us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
W20260927 21:14:53.423411 15384 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33899 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33899: Client connection negotiation failed: client connection to 127.0.98.62:33899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:14:53.423761 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.423639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:60397)
0927 21:14:53.423686 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.423740 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 21:14:53.424077 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.423989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:44187)
0927 21:14:53.424030 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.424054 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 21:14:53.424182 15384 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:14:53.427688 15388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:53.427896 15388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:53.427920 15388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:53.429117 15388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:53.429159 15388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:53.429186 15388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:14:53.430837 15388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37475
--tserver_master_addrs=127.0.98.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:53.431049 15388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:53.431246 15388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:53.436172 15397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:53.436333 15388 server_base.cc:1034] running on GCE node
W20260927 21:14:53.436187 15395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:53.436184 15394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:53.436623 15388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:53.436831 15388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:53.437955 15388 hybrid_clock.cc:648] HybridClock initialized: now 1790543693437940 us; error 30 us; skew 500 ppm
Sep 27 21:14:53 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:53.440362 15388 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:14:53.440775 15388 webserver.cc:460] Webserver started at http://127.0.98.2:37475/ using document root <none> and password file <none>
I20260927 21:14:53.440996 15388 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:53.441046 15388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:53.442373 15388 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:53.442895 15404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:53.443044 15388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:14:53.443123 15388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "f5f74a8a0d484781807632e23bdea3eb"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "07b019ac1b512126c8f5f081f1ad1584"
server_key_iv: "2435e2161c306ff0aff590ed53fbf963"
server_key_version: "encryptionkey@0"
I20260927 21:14:53.443449 15388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:53.472764 15388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:53.474869 15388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:53.475024 15388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:53.475241 15388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:53.475529 15388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:53.475564 15388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:53.475594 15388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:53.475611 15388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:53.481221 15388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44295
I20260927 21:14:53.481330 15517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44295 every 8 connection(s)
I20260927 21:14:53.481644 15388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:14:53.482689 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.482290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:36469)
0927 21:14:53.482458 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.482542 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
W20260927 21:14:53.483078 15518 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33899 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33899: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
I20260927 21:14:53.483336 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.483254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:38005)
0927 21:14:53.483291 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.483318 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 21:14:53.483541 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.483484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:41575)
0927 21:14:53.483510 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.483528 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 21:14:53.483620 15518 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:14:53.490366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 15388
I20260927 21:14:53.490623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45989
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33899
--webserver_port=35475
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:14:53.573770 15521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:14:53.574011 15521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:14:53.574038 15521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:14:53.575414 15521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:14:53.575470 15521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:14:53.575500 15521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:14:53.577167 15521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35475
--tserver_master_addrs=127.0.98.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:14:53.577481 15521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:14:53.577711 15521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:14:53.582813 15529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:53.582835 15526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:14:53.582875 15527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:14:53.583142 15521 server_base.cc:1034] running on GCE node
I20260927 21:14:53.583333 15521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:14:53.583572 15521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:14:53.584718 15521 hybrid_clock.cc:648] HybridClock initialized: now 1790543693584695 us; error 35 us; skew 500 ppm
Sep 27 21:14:53 dist-test-slave-r9x1 krb5kdc[14083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:14:53.587224 15521 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:14:53.587695 15521 webserver.cc:460] Webserver started at http://127.0.98.3:35475/ using document root <none> and password file <none>
I20260927 21:14:53.587915 15521 fs_manager.cc:359] Metadata directory not provided
I20260927 21:14:53.587965 15521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:14:53.589248 15521 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:14:53.589915 15536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:53.590091 15521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:14:53.590176 15521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "47af539ed94c47eba5b6bd07687464b6"
format_stamp: "Formatted at 2026-09-27 21:14:51 on dist-test-slave-r9x1"
server_key: "c787cff65469c395d19c57ef294b4237"
server_key_iv: "cc33550399d9297903ece970f18623e6"
server_key_version: "encryptionkey@0"
I20260927 21:14:53.590572 15521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:14:53.632082 15521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:14:53.634223 15521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:14:53.634359 15521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:14:53.634585 15521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:14:53.634886 15521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:14:53.634919 15521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:53.634949 15521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:14:53.634971 15521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:14:53.641396 15521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45989
I20260927 21:14:53.641470 15649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45989 every 8 connection(s)
I20260927 21:14:53.641803 15521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:14:53.642884 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.642487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:47095)
0927 21:14:53.642665 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.642740 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
W20260927 21:14:53.643287 15650 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33899 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33899: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
I20260927 21:14:53.643580 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.643488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:53211)
0927 21:14:53.643532 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.643558 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 21:14:53.643843 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:53.643768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:53013)
0927 21:14:53.643803 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:53.643824 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 21:14:53.643931 15650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:14:53.646189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 15521
I20260927 21:14:54.424863 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:54.424592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:59509)
0927 21:14:54.424747 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:54.424795 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260927 21:14:54.484264 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:54.484072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:41587)
0927 21:14:54.484177 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:54.484205 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260927 21:14:54.644466 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:54.644233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:55923)
0927 21:14:54.644371 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:54.644410 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260927 21:14:54.747503 15657 raft_consensus.cc:482] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:14:54.747627 15657 raft_consensus.cc:504] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:54.747907 15657 leader_election.cc:288] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295), 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989)
Sep 27 21:14:54 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 21:14:54 dist-test-slave-r9x1 krb5kdc[14083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 21:14:54.760366 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:54.748251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58411 (local address 127.0.98.2:44295)
0927 21:14:54.748425 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:54.748432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:54.748461 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:14:54.749014 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:54.749017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:54.749080 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:54.749227 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:54.750245 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:54.751670 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:54.753290 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:54.753944 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:54.754068 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:54.756983 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:54.756987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:54.756988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:54.757000 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:54.758643 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:54.759114 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:54.759118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:54.759118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:54.759215 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:54.759497 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:54.759499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:54.759499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:54.759765 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:54.759840 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:54.760119 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 21:14:54.760249 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":59,"threads_started":1}
I20260927 21:14:54.760377 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:54.748166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49261 (local address 127.0.98.3:45989)
0927 21:14:54.748343 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:14:54.748355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:14:54.748390 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:14:54.749017 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:14:54.749020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:14:54.749080 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:14:54.749227 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:14:54.750251 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:54.751660 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:54.753291 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:14:54.753945 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:14:54.754064 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:54.756676 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:14:54.756681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:14:54.756682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:14:54.756697 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:14:54.758586 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:54.759152 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:54.759155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:54.759156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:54.759276 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:14:54.759509 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:14:54.759510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:14:54.759511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:14:54.759766 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:14:54.759836 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:14:54.760121 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 21:14:54.760246 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260927 21:14:54.760375 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:54.748089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44295 (local address 127.0.98.1:58411)
0927 21:14:54.748354 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:54.748365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:54.748420 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:54.749342 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:54.749343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:54.749825 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:54.750117 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:54.750125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:54.751812 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:54.751813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:54.753171 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:54.753175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:54.754044 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:54.754045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:54.754196 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:54.754796 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:54.754800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:54.756851 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:54.758741 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:54.758742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:54.758742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:54.759024 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:54.759299 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:54.759300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:54.759301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:54.759427 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:54.759843 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:54.759844 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:54.759960 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 21:14:54.760050 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:14:54.760258 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":255,"thread_start_us":55,"threads_started":1}
I20260927 21:14:54.760699 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:54.748089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45989 (local address 127.0.98.1:49261)
0927 21:14:54.748195 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:54.748209 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:14:54.748285 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:14:54.749326 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:14:54.749330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:14:54.749825 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:14:54.750117 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:54.750125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:54.751797 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:54.751799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:54.753171 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:14:54.753175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:14:54.754042 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:14:54.754044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:14:54.754196 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:14:54.754789 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:14:54.754800 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:14:54.756539 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:14:54.758689 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:54.758699 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:54.758702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:54.759055 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:54.759347 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:14:54.759348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:14:54.759348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:14:54.759431 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:14:54.759833 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:14:54.759840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:14:54.759964 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 21:14:54.760051 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:14:54.760333 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":195}
W20260927 21:14:54.761492 15271 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
W20260927 21:14:54.761584 15270 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
I20260927 21:14:54.761648 15270 leader_election.cc:302] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14859606b1ac4cf280ba942361bf0fa0; no voters: 47af539ed94c47eba5b6bd07687464b6, f5f74a8a0d484781807632e23bdea3eb
I20260927 21:14:54.761755 15657 raft_consensus.cc:2726] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:14:55.425755 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:55.425473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:33815)
0927 21:14:55.425656 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:55.425705 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260927 21:14:55.484987 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:55.484743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:42179)
0927 21:14:55.484892 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:55.484941 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260927 21:14:55.645139 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:55.644902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:33995)
0927 21:14:55.645038 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:55.645084 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260927 21:14:56.426550 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:56.426301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:40703)
0927 21:14:56.426457 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:56.426502 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260927 21:14:56.440593 15665 raft_consensus.cc:482] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:14:56.440699 15665 raft_consensus.cc:504] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:56.440905 15665 leader_election.cc:288] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295), 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989)
W20260927 21:14:56.441322 15271 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
W20260927 21:14:56.441399 15270 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
I20260927 21:14:56.441473 15270 leader_election.cc:302] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14859606b1ac4cf280ba942361bf0fa0; no voters: 47af539ed94c47eba5b6bd07687464b6, f5f74a8a0d484781807632e23bdea3eb
I20260927 21:14:56.441557 15665 raft_consensus.cc:2726] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:14:56.485785 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:56.485553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:50359)
0927 21:14:56.485697 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:56.485732 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260927 21:14:56.645789 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:56.645583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:57065)
0927 21:14:56.645702 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:56.645744 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260927 21:14:57.427335 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:57.427102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:51353)
0927 21:14:57.427231 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:57.427285 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260927 21:14:57.486423 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:57.486220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:56145)
0927 21:14:57.486338 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:57.486373 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260927 21:14:57.646486 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:57.646220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:48151)
0927 21:14:57.646395 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:57.646438 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":68,"threads_started":1}
I20260927 21:14:58.428196 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:58.427914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:45407)
0927 21:14:58.428099 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:58.428146 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260927 21:14:58.487115 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:58.486873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:39939)
0927 21:14:58.487021 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:58.487067 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260927 21:14:58.647308 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:58.647061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:42027)
0927 21:14:58.647214 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:58.647260 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260927 21:14:59.428952 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:59.428742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:43401)
0927 21:14:59.428860 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:59.428903 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260927 21:14:59.487866 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:59.487632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:40849)
0927 21:14:59.487763 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:59.487814 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260927 21:14:59.648099 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:14:59.647845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:41131)
0927 21:14:59.647997 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:14:59.648050 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260927 21:14:59.661798 15677 raft_consensus.cc:482] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:14:59.661921 15677 raft_consensus.cc:504] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:14:59.662096 15677 leader_election.cc:288] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295), 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989)
W20260927 21:14:59.662426 15271 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
W20260927 21:14:59.662495 15270 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
I20260927 21:14:59.662532 15270 leader_election.cc:302] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14859606b1ac4cf280ba942361bf0fa0; no voters: 47af539ed94c47eba5b6bd07687464b6, f5f74a8a0d484781807632e23bdea3eb
I20260927 21:14:59.662623 15677 raft_consensus.cc:2726] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:15:00.429754 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:00.429519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:44135)
0927 21:15:00.429660 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:00.429706 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260927 21:15:00.488642 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:00.488390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:44319)
0927 21:15:00.488531 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:00.488579 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260927 21:15:00.649046 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:00.648685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:54389)
0927 21:15:00.648921 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:00.648992 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
I20260927 21:15:01.430678 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:01.430309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:60459)
0927 21:15:01.430528 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:01.430603 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260927 21:15:01.489467 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:01.489175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:56651)
0927 21:15:01.489368 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:01.489417 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":130,"threads_started":1}
I20260927 21:15:01.639312 15683 raft_consensus.cc:482] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:15:01.639398 15683 raft_consensus.cc:504] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:15:01.639569 15683 leader_election.cc:288] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295), 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989)
W20260927 21:15:01.639935 15271 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
W20260927 21:15:01.639998 15270 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
I20260927 21:15:01.640084 15270 leader_election.cc:302] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14859606b1ac4cf280ba942361bf0fa0; no voters: 47af539ed94c47eba5b6bd07687464b6, f5f74a8a0d484781807632e23bdea3eb
I20260927 21:15:01.640151 15683 raft_consensus.cc:2726] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:15:01.649885 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:01.649648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:60117)
0927 21:15:01.649781 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:01.649819 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260927 21:15:02.431651 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:02.431399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:46833)
0927 21:15:02.431549 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:02.431600 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260927 21:15:02.490325 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:02.490068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:39935)
0927 21:15:02.490221 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:02.490272 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260927 21:15:02.650709 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:02.650459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:45747)
0927 21:15:02.650606 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:02.650654 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260927 21:15:03.432512 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:03.432276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:41643)
0927 21:15:03.432416 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:03.432463 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260927 21:15:03.491200 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:03.490937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:33539)
0927 21:15:03.491101 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:03.491150 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260927 21:15:03.651554 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:03.651279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:34183)
0927 21:15:03.651458 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:03.651505 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260927 21:15:04.433364 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:04.433088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:34337)
0927 21:15:04.433233 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:04.433310 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260927 21:15:04.491858 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:04.491619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:45127)
0927 21:15:04.491770 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:04.491808 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
I20260927 21:15:04.652645 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:04.652277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:39199)
0927 21:15:04.652526 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:04.652590 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":127,"threads_started":1}
I20260927 21:15:05.434119 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:05.433856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:43513)
0927 21:15:05.434011 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:05.434064 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260927 21:15:05.492645 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:05.492430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:60659)
0927 21:15:05.492555 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:05.492596 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260927 21:15:05.653484 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:05.653238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:52233)
0927 21:15:05.653398 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:05.653432 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260927 21:15:06.434962 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:06.434693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:60467)
0927 21:15:06.434867 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:06.434914 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260927 21:15:06.493563 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:06.493321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:49267)
0927 21:15:06.493467 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:06.493511 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260927 21:15:06.654289 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:06.654010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:54175)
0927 21:15:06.654187 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:06.654240 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260927 21:15:07.435855 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:07.435572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:59023)
0927 21:15:07.435754 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:07.435804 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260927 21:15:07.494379 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:07.494119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:58757)
0927 21:15:07.494274 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:07.494328 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260927 21:15:07.655229 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:07.654908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:57131)
0927 21:15:07.655122 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:07.655171 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260927 21:15:08.069692 15703 raft_consensus.cc:482] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:15:08.069799 15703 raft_consensus.cc:504] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f74a8a0d484781807632e23bdea3eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "47af539ed94c47eba5b6bd07687464b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45989 } } peers { permanent_uuid: "14859606b1ac4cf280ba942361bf0fa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44677 } }
I20260927 21:15:08.069978 15703 leader_election.cc:288] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295), 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989)
W20260927 21:15:08.070374 15270 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5f74a8a0d484781807632e23bdea3eb (127.0.98.2:44295): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
W20260927 21:15:08.070459 15271 leader_election.cc:341] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47af539ed94c47eba5b6bd07687464b6 (127.0.98.3:45989): Not found: Tablet not found: 48e147d86d8c44fda29f17f6595815d5
I20260927 21:15:08.070549 15271 leader_election.cc:302] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14859606b1ac4cf280ba942361bf0fa0; no voters: 47af539ed94c47eba5b6bd07687464b6, f5f74a8a0d484781807632e23bdea3eb
I20260927 21:15:08.070621 15703 raft_consensus.cc:2726] T 48e147d86d8c44fda29f17f6595815d5 P 14859606b1ac4cf280ba942361bf0fa0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:15:08.436784 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:08.436540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.1:54441)
0927 21:15:08.436690 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:08.436734 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260927 21:15:08.495324 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:08.495043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.2:38135)
0927 21:15:08.495223 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:08.495274 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":59,"threads_started":1}
I20260927 21:15:08.650698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 15225
I20260927 21:15:08.655490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 15388
I20260927 21:15:08.657497 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:08.657215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33899 (local address 127.0.98.3:50955)
0927 21:15:08.657392 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:08.657445 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260927 21:15:08.660301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 15521
2026-09-27T21:15:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18504 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:15:08 dist-test-slave-r9x1 krb5kdc[15709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:15:08 dist-test-slave-r9x1 krb5kdc[15709](info): set up 2 sockets
Sep 27 21:15:08 dist-test-slave-r9x1 krb5kdc[15709](info): commencing operation
krb5kdc: starting...
W20260927 21:15:09.621088 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:15:09 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:15:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:15:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:15:09.744082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:15:09.827787 15722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:09.828017 15722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:09.828050 15722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:09.829806 15722 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:15:09.829874 15722 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:15:09.829895 15722 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:15:09.829909 15722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:15:09.829928 15722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:09.829955 15722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:09.829972 15722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:15:09.829993 15722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:15:09.832267 15722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:09.832700 15722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:09.832983 15722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:15:09.838191 15727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:09.838213 15730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:09.838254 15722 server_base.cc:1034] running on GCE node
W20260927 21:15:09.838213 15728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:09.838693 15722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:09.838986 15722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:09.840165 15722 hybrid_clock.cc:648] HybridClock initialized: now 1790543709840148 us; error 30 us; skew 500 ppm
Sep 27 21:15:09 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:09.842789 15722 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:15:09.843271 15722 webserver.cc:460] Webserver started at http://127.0.98.62:34247/ using document root <none> and password file <none>
I20260927 21:15:09.843492 15722 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:09.843547 15722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:09.843676 15722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:15:09.844775 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "ab41038cbe4648159c32b0c6b7c130dd"
format_stamp: "Formatted at 2026-09-27 21:15:09 on dist-test-slave-r9x1"
server_key: "75d241e5d621ad68d8465c4f26c9ef4e"
server_key_iv: "4c0c551ac27d9494c34e6ee7c377b6cc"
server_key_version: "encryptionkey@0"
I20260927 21:15:09.845113 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab41038cbe4648159c32b0c6b7c130dd"
format_stamp: "Formatted at 2026-09-27 21:15:09 on dist-test-slave-r9x1"
server_key: "75d241e5d621ad68d8465c4f26c9ef4e"
server_key_iv: "4c0c551ac27d9494c34e6ee7c377b6cc"
server_key_version: "encryptionkey@0"
I20260927 21:15:09.846524 15722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:15:09.847297 15737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:09.847457 15722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:09.847576 15722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "ab41038cbe4648159c32b0c6b7c130dd"
format_stamp: "Formatted at 2026-09-27 21:15:09 on dist-test-slave-r9x1"
server_key: "75d241e5d621ad68d8465c4f26c9ef4e"
server_key_iv: "4c0c551ac27d9494c34e6ee7c377b6cc"
server_key_version: "encryptionkey@0"
I20260927 21:15:09.847644 15722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:09.875837 15722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:09.877942 15722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:09.878089 15722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:09.882028 15722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46331
I20260927 21:15:09.882061 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46331 every 8 connection(s)
I20260927 21:15:09.882407 15722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:15:09.883282 15790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:15:09.885843 15790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Bootstrap starting.
I20260927 21:15:09.886443 15790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Neither blocks nor log segments found. Creating new log.
I20260927 21:15:09.886693 15790 log.cc:824] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Log is configured to *not* fsync() on all Append() calls
I20260927 21:15:09.887492 15790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: No bootstrap required, opened a new log
I20260927 21:15:09.889434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 15722
I20260927 21:15:09.889520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:15:09.889498 15790 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:09.889695 392 external_mini_cluster.cc:1428] Setting key 5ff86bcffc0b8742f26c76650ce3c564
I20260927 21:15:09.889701 15790 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:15:09.889734 15790 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab41038cbe4648159c32b0c6b7c130dd, State: Initialized, Role: FOLLOWER
I20260927 21:15:09.889895 15790 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:09.889979 15790 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:15:09.890007 15790 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:15:09.890086 15790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:15:09.891301 15790 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:09.891438 15790 leader_election.cc:302] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab41038cbe4648159c32b0c6b7c130dd; no voters:
I20260927 21:15:09.891657 15790 leader_election.cc:288] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:15:09.891685 15795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:15:09.891785 15795 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 1 LEADER]: Becoming Leader. State: Replica: ab41038cbe4648159c32b0c6b7c130dd, State: Running, Role: LEADER
I20260927 21:15:09.892036 15795 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:09.892041 15790 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:15:09.892468 15796 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab41038cbe4648159c32b0c6b7c130dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } } }
I20260927 21:15:09.892462 15797 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab41038cbe4648159c32b0c6b7c130dd. Latest consensus state: current_term: 1 leader_uuid: "ab41038cbe4648159c32b0c6b7c130dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } } }
I20260927 21:15:09.892659 15796 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: This master's current role is: LEADER
I20260927 21:15:09.892666 15797 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: This master's current role is: LEADER
I20260927 21:15:09.893426 15804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:15:09.894778 15804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:15:09.896481 15804 catalog_manager.cc:1347] Generated new cluster ID: cc2256f79cbc4c18a37f403902859a04
I20260927 21:15:09.896530 15804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:15:09 dist-test-slave-r9x1 krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:15:09.902415 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:09.890137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.0.98.62:46331)
0927 21:15:09.890353 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:09.890361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:09.890473 (+ 112us) server_negotiation.cc:407] Connection header received
0927 21:15:09.890688 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:09.890694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:09.890742 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:09.890856 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:09.891648 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:09.893187 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:09.894108 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:09.894750 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:09.894869 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:09.898154 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:09.898158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:09.898159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:09.898173 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:09.900594 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:09.901059 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:09.901062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:09.901063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:09.901179 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:09.901424 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:09.901425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:09.901425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:09.901752 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:09.901878 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:09.902033 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 21:15:09.902179 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260927 21:15:09.905964 15804 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:15:09.906750 15804 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:15:09.919390 15804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Generated new TSK 0
I20260927 21:15:09.919651 15804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:15:09.953122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46331
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:15:10.036607 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:10.036841 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:10.036868 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:10.038193 15816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:10.038247 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:10.038276 15816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:15:10.039897 15816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:10.040153 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:10.040392 15816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:15:10.045718 15824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:10.045718 15821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:10.045714 15822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:10.046023 15816 server_base.cc:1034] running on GCE node
I20260927 21:15:10.046277 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:10.046543 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:10.047742 15816 hybrid_clock.cc:648] HybridClock initialized: now 1790543710047692 us; error 76 us; skew 500 ppm
Sep 27 21:15:10 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:10.050635 15816 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:15:10.051120 15816 webserver.cc:460] Webserver started at http://127.0.98.1:36897/ using document root <none> and password file <none>
I20260927 21:15:10.051308 15816 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:10.051347 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:10.051455 15816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:15:10.052546 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ea660de4d814fad87e39ef67dca8521"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "d774d30e4a018334e3906976c98dc59b"
server_key_iv: "76a59784715e468c2043874b994ae589"
server_key_version: "encryptionkey@0"
I20260927 21:15:10.052860 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ea660de4d814fad87e39ef67dca8521"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "d774d30e4a018334e3906976c98dc59b"
server_key_iv: "76a59784715e468c2043874b994ae589"
server_key_version: "encryptionkey@0"
I20260927 21:15:10.055182 15816 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:15:10.056627 15831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:10.056926 15816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:10.057065 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "5ea660de4d814fad87e39ef67dca8521"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "d774d30e4a018334e3906976c98dc59b"
server_key_iv: "76a59784715e468c2043874b994ae589"
server_key_version: "encryptionkey@0"
I20260927 21:15:10.057189 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:10.064925 15816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:10.066972 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:10.067111 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:10.067343 15816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:15:10.067656 15816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:15:10.067690 15816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:10.067719 15816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:15:10.067746 15816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:10.073537 15816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44231
I20260927 21:15:10.073585 15944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44231 every 8 connection(s)
I20260927 21:15:10.073904 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:15:10.078361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 15816
I20260927 21:15:10.078467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:15:10.078680 392 external_mini_cluster.cc:1428] Setting key fd5ef924602ba91ec9ba435ce3a7efb1
Sep 27 21:15:10 dist-test-slave-r9x1 krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:15:10.083796 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.074654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39229 (local address 127.0.98.62:46331)
0927 21:15:10.074710 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.074711 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.074907 (+ 196us) server_negotiation.cc:407] Connection header received
0927 21:15:10.075391 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.075392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.075406 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.075463 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:10.076393 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.076893 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.077928 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.078139 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.078209 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.080679 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:10.080681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:10.080681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:10.080696 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:10.082357 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:10.082824 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:10.082826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:10.082826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:10.082894 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:10.083144 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:10.083145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:10.083146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:10.083296 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:10.083364 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.083646 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.083722 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:15:10.083953 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.074620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:39229)
0927 21:15:10.074781 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:10.074824 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:10.075314 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:10.075525 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:10.075529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:10.076099 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:15:10.076311 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.076316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.076963 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.076964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.077842 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.077845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.078217 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.078218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.078319 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.078911 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:15:10.078922 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:15:10.080538 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:15:10.082465 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:10.082475 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:10.082477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:10.082743 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:10.082963 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:10.082964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:10.082964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:10.083082 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:10.083360 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:15:10.083366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:15:10.083516 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 21:15:10.083611 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:15:10.083751 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260927 21:15:10.084481 15945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46331
I20260927 21:15:10.084584 15945 heartbeater.cc:464] Registering TS with master...
I20260927 21:15:10.084860 15945 heartbeater.cc:511] Master 127.0.98.62:46331 requested a full tablet report, sending...
I20260927 21:15:10.085542 15754 ts_manager.cc:194] Registered new tserver with Master: 5ea660de4d814fad87e39ef67dca8521 (127.0.98.1:44231)
I20260927 21:15:10.086488 15754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39229
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:15:10.106227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46331
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:15:10.189956 15950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:10.190189 15950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:10.190215 15950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:10.191478 15950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:10.191527 15950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:10.191556 15950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:15:10.193177 15950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:10.193467 15950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:10.193694 15950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:15:10.198858 15958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:10.198894 15950 server_base.cc:1034] running on GCE node
W20260927 21:15:10.198863 15956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:10.198915 15955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:10.199328 15950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:10.199558 15950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:10.200795 15950 hybrid_clock.cc:648] HybridClock initialized: now 1790543710200779 us; error 33 us; skew 500 ppm
Sep 27 21:15:10 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:10.203605 15950 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:15:10.204073 15950 webserver.cc:460] Webserver started at http://127.0.98.2:38683/ using document root <none> and password file <none>
I20260927 21:15:10.204303 15950 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:10.204348 15950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:10.204471 15950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:15:10.205670 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "a58fbe5275644d39aa760b196c290609"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "c9ccdc55a1bdd728426bca2365244613"
server_key_iv: "42254ee8e47220a40e1ed43d8754278f"
server_key_version: "encryptionkey@0"
I20260927 21:15:10.206053 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a58fbe5275644d39aa760b196c290609"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "c9ccdc55a1bdd728426bca2365244613"
server_key_iv: "42254ee8e47220a40e1ed43d8754278f"
server_key_version: "encryptionkey@0"
I20260927 21:15:10.207499 15950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:15:10.208302 15965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:10.208490 15950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:10.208577 15950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "a58fbe5275644d39aa760b196c290609"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "c9ccdc55a1bdd728426bca2365244613"
server_key_iv: "42254ee8e47220a40e1ed43d8754278f"
server_key_version: "encryptionkey@0"
I20260927 21:15:10.208640 15950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:10.224781 15950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:10.227008 15950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:10.227130 15950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:10.227330 15950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:15:10.227633 15950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:15:10.227670 15950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:10.227694 15950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:15:10.227718 15950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:10.233317 15950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46619
I20260927 21:15:10.233377 16078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46619 every 8 connection(s)
I20260927 21:15:10.233688 15950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:15:10 dist-test-slave-r9x1 krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:15:10.241511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 15950
I20260927 21:15:10.241597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:15:10.241783 392 external_mini_cluster.cc:1428] Setting key e3e6f67f8b97fd026841e0094f0e6c39
I20260927 21:15:10.244252 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.234463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54673 (local address 127.0.98.62:46331)
0927 21:15:10.234529 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.234531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.234777 (+ 246us) server_negotiation.cc:407] Connection header received
0927 21:15:10.235306 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.235307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.235321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.235367 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:10.236366 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.236887 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.237926 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.238132 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.238192 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.240481 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:10.240483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:10.240484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:10.240492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:10.242528 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:10.243108 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:10.243109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:10.243109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:10.243171 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:10.243483 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:10.243484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:10.243485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:10.243618 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:10.243689 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.244039 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.244149 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:15:10.244455 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.234430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:54673)
0927 21:15:10.234617 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:10.234682 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:10.235203 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:10.235454 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:10.235458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:10.236059 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:15:10.236260 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.236264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.236981 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.236982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.237830 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.237833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.238212 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.238213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.238344 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.238926 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:15:10.238935 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:15:10.240369 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:15:10.242635 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:10.242650 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:10.242652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:10.243026 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:10.243236 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:10.243236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:10.243237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:10.243402 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:10.243694 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:15:10.243702 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:15:10.243895 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 21:15:10.244011 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:15:10.244231 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
I20260927 21:15:10.244944 16079 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46331
I20260927 21:15:10.245059 16079 heartbeater.cc:464] Registering TS with master...
I20260927 21:15:10.245250 16079 heartbeater.cc:511] Master 127.0.98.62:46331 requested a full tablet report, sending...
I20260927 21:15:10.245704 15754 ts_manager.cc:194] Registered new tserver with Master: a58fbe5275644d39aa760b196c290609 (127.0.98.2:46619)
I20260927 21:15:10.246192 15754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54673
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:15:10.269083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46331
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:15:10.353580 16084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:10.353806 16084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:10.353832 16084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:10.355157 16084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:10.355216 16084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:10.355245 16084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:15:10.356842 16084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:10.357082 16084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:10.357309 16084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:15:10.363497 16084 server_base.cc:1034] running on GCE node
W20260927 21:15:10.363389 16089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:10.363431 16090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:10.363411 16092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:10.363932 16084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:10.364207 16084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:10.365340 16084 hybrid_clock.cc:648] HybridClock initialized: now 1790543710365329 us; error 35 us; skew 500 ppm
Sep 27 21:15:10 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:10.367924 16084 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:15:10.368438 16084 webserver.cc:460] Webserver started at http://127.0.98.3:43949/ using document root <none> and password file <none>
I20260927 21:15:10.368646 16084 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:10.368686 16084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:10.368788 16084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:15:10.369925 16084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "66b910ab3ea94b90afd5aaac58169fba"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "a74488d3119bd6ba5737dfd698e92dd3"
server_key_iv: "62f2aab0ac0573ab5653a47d368bc8fb"
server_key_version: "encryptionkey@0"
I20260927 21:15:10.370234 16084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66b910ab3ea94b90afd5aaac58169fba"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "a74488d3119bd6ba5737dfd698e92dd3"
server_key_iv: "62f2aab0ac0573ab5653a47d368bc8fb"
server_key_version: "encryptionkey@0"
I20260927 21:15:10.371565 16084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:15:10.372331 16099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:10.372509 16084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:10.372592 16084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "66b910ab3ea94b90afd5aaac58169fba"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "a74488d3119bd6ba5737dfd698e92dd3"
server_key_iv: "62f2aab0ac0573ab5653a47d368bc8fb"
server_key_version: "encryptionkey@0"
I20260927 21:15:10.372664 16084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:10.391217 16084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:10.393630 16084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:10.393784 16084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:10.394016 16084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:15:10.394326 16084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:15:10.394361 16084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:10.394389 16084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:15:10.394413 16084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:10.400691 16084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36705
I20260927 21:15:10.400784 16212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36705 every 8 connection(s)
I20260927 21:15:10.401413 16084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:15:10.404225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 16084
I20260927 21:15:10.404340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:15:10.404526 392 external_mini_cluster.cc:1428] Setting key 8d6ea2f93bb1fc907d1df5fcb2c307f9
Sep 27 21:15:10 dist-test-slave-r9x1 krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:15:10.413175 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.402199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44495 (local address 127.0.98.62:46331)
0927 21:15:10.402276 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.402278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.402454 (+ 176us) server_negotiation.cc:407] Connection header received
0927 21:15:10.402985 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.402990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.403002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.403066 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:10.404071 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.404866 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.406393 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.406627 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.406692 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.409602 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:10.409606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:10.409606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:10.409619 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:10.411584 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:10.412107 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:10.412108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:10.412109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:10.412177 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:10.412470 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:10.412470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:10.412471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:10.412626 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:10.412678 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.413002 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.413091 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:15:10.413353 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.402147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:44495)
0927 21:15:10.402318 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:10.402365 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:10.402868 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:10.403123 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:10.403126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:10.403734 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:15:10.403944 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.403949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.405003 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.405005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.406279 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.406284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.406712 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.406728 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.406846 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.407393 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:15:10.407404 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:15:10.408974 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:15:10.411700 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:10.411710 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:10.411713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:10.412018 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:10.412264 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:10.412265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:10.412265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:10.412398 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:10.412680 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:15:10.412686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:15:10.412838 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 21:15:10.412945 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:15:10.413097 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260927 21:15:10.413837 16213 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46331
I20260927 21:15:10.413933 16213 heartbeater.cc:464] Registering TS with master...
I20260927 21:15:10.414103 16213 heartbeater.cc:511] Master 127.0.98.62:46331 requested a full tablet report, sending...
I20260927 21:15:10.414438 15754 ts_manager.cc:194] Registered new tserver with Master: 66b910ab3ea94b90afd5aaac58169fba (127.0.98.3:36705)
I20260927 21:15:10.414788 15754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44495
I20260927 21:15:10.416757 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:15:10.424317 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.418071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.0.98.62:46331)
0927 21:15:10.418125 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.418127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.418238 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:15:10.418285 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.418286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.418295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.418342 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:10.418902 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.419564 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.420355 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.420555 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.420616 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.421908 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:10.421909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:10.421910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:10.421917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:10.423259 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:10.423637 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:10.423638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:10.423638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:10.423686 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:10.423823 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:10.423824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:10.423824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:10.423971 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:10.424043 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.424183 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.424247 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:15:10.426520 15754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:15:10.428237 15754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:15:10.434954 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.430836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44231 (local address 127.0.0.1:54688)
0927 21:15:10.431252 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:10.431261 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:10.431310 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:10.431523 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:10.431524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:10.431526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:10.431572 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.431573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.432585 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.432586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.433909 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.433912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.434619 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.434619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.434711 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.434726 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:15:10.434787 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:15:10.434847 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":114,"threads_started":1}
I20260927 21:15:10.435076 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.431020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.0.98.1:44231)
0927 21:15:10.431211 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.431218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.431315 (+ 97us) server_negotiation.cc:407] Connection header received
0927 21:15:10.431362 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.431364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.431377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.431451 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:10.431665 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.432496 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.434001 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.434540 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.434604 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.434668 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.434947 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.434995 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260927 21:15:10.435158 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.430833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.0.1:55704)
0927 21:15:10.431373 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:10.431381 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:10.431413 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:10.431852 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:10.431853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:10.431855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:10.431898 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.431900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.433071 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.433072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.434006 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.434009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.434940 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.434941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.434991 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.434994 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:15:10.435033 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 21:15:10.435077 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260927 21:15:10.435397 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.431201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.0.98.3:36705)
0927 21:15:10.431703 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.431707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.431716 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:15:10.431750 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.431751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.431766 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.431843 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:10.431962 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.432986 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.434074 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.434790 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.434859 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.434912 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.435260 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.435314 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":91,"threads_started":1}
I20260927 21:15:10.435638 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.430833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46619 (local address 127.0.0.1:38376)
0927 21:15:10.431026 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:10.431071 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:10.431154 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:10.431483 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:10.431486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:10.431488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:10.431565 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.431567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.432882 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.432883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.433960 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.433962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.434887 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.434888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.434980 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.434984 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:15:10.435540 (+ 556us) client_negotiation.cc:240] Negotiation successful
0927 21:15:10.435586 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":61,"threads_started":1}
I20260927 21:15:10.435781 15879 tablet_service.cc:1467] Processing CreateTablet for tablet 1dadfbd08d104549842ac358a662860c (DEFAULT_TABLE table=test-table [id=1dde208efa364fb795c659bb6eeb5cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:15:10.436020 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.431077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.0.98.2:46619)
0927 21:15:10.431254 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.431258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.431267 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:15:10.431306 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.431308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.431324 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.431414 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:10.431648 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.432775 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.434043 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.434788 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.435802 (+ 1014us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.435876 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.435902 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.435939 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260927 21:15:10.435992 16147 tablet_service.cc:1467] Processing CreateTablet for tablet 1dadfbd08d104549842ac358a662860c (DEFAULT_TABLE table=test-table [id=1dde208efa364fb795c659bb6eeb5cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:15:10.436177 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dadfbd08d104549842ac358a662860c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:15:10.436275 16147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dadfbd08d104549842ac358a662860c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:15:10.436636 16013 tablet_service.cc:1467] Processing CreateTablet for tablet 1dadfbd08d104549842ac358a662860c (DEFAULT_TABLE table=test-table [id=1dde208efa364fb795c659bb6eeb5cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:15:10.436895 16013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dadfbd08d104549842ac358a662860c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:15:10.439726 16230 tablet_bootstrap.cc:492] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: Bootstrap starting.
I20260927 21:15:10.439993 16231 tablet_bootstrap.cc:492] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Bootstrap starting.
I20260927 21:15:10.440207 16232 tablet_bootstrap.cc:492] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: Bootstrap starting.
I20260927 21:15:10.440469 16230 tablet_bootstrap.cc:654] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: Neither blocks nor log segments found. Creating new log.
I20260927 21:15:10.440865 16232 tablet_bootstrap.cc:654] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: Neither blocks nor log segments found. Creating new log.
I20260927 21:15:10.441053 16230 log.cc:824] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: Log is configured to *not* fsync() on all Append() calls
I20260927 21:15:10.441406 16232 log.cc:824] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: Log is configured to *not* fsync() on all Append() calls
I20260927 21:15:10.441903 16231 tablet_bootstrap.cc:654] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Neither blocks nor log segments found. Creating new log.
I20260927 21:15:10.441928 16230 tablet_bootstrap.cc:492] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: No bootstrap required, opened a new log
I20260927 21:15:10.441984 16230 ts_tablet_manager.cc:1397] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:15:10.442147 16232 tablet_bootstrap.cc:492] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: No bootstrap required, opened a new log
I20260927 21:15:10.442219 16232 ts_tablet_manager.cc:1397] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:15:10.442456 16231 log.cc:824] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Log is configured to *not* fsync() on all Append() calls
I20260927 21:15:10.443207 16231 tablet_bootstrap.cc:492] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: No bootstrap required, opened a new log
I20260927 21:15:10.443280 16231 ts_tablet_manager.cc:1397] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:15:10.443765 16230 raft_consensus.cc:348] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:10.443924 16230 raft_consensus.cc:374] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:15:10.443956 16230 raft_consensus.cc:729] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58fbe5275644d39aa760b196c290609, State: Initialized, Role: FOLLOWER
I20260927 21:15:10.444100 16230 consensus_queue.cc:260] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:10.444376 16230 ts_tablet_manager.cc:1428] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 21:15:10.444253 16232 raft_consensus.cc:348] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:10.444416 16232 raft_consensus.cc:374] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:15:10.444458 16232 raft_consensus.cc:729] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b910ab3ea94b90afd5aaac58169fba, State: Initialized, Role: FOLLOWER
I20260927 21:15:10.444490 16079 heartbeater.cc:503] Master 127.0.98.62:46331 was elected leader, sending a full tablet report...
I20260927 21:15:10.444568 16232 consensus_queue.cc:260] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:10.444842 16232 ts_tablet_manager.cc:1428] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:15:10.445327 16213 heartbeater.cc:503] Master 127.0.98.62:46331 was elected leader, sending a full tablet report...
I20260927 21:15:10.445257 16231 raft_consensus.cc:348] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:10.445459 16231 raft_consensus.cc:374] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:15:10.445519 16231 raft_consensus.cc:729] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea660de4d814fad87e39ef67dca8521, State: Initialized, Role: FOLLOWER
I20260927 21:15:10.445628 16231 consensus_queue.cc:260] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:10.445901 16231 ts_tablet_manager.cc:1428] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:15:10.445926 15945 heartbeater.cc:503] Master 127.0.98.62:46331 was elected leader, sending a full tablet report...
W20260927 21:15:10.484311 16080 tablet.cc:2369] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:15:10.574659 15946 tablet.cc:2369] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:15:10.652029 16214 tablet.cc:2369] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:15:10.722966 16237 raft_consensus.cc:482] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:15:10.723099 16237 raft_consensus.cc:504] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:10.723466 16237 leader_election.cc:288] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a58fbe5275644d39aa760b196c290609 (127.0.98.2:46619), 5ea660de4d814fad87e39ef67dca8521 (127.0.98.1:44231)
I20260927 21:15:10.726434 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.723748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48753 (local address 127.0.98.2:46619)
0927 21:15:10.723797 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.723799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.723927 (+ 128us) server_negotiation.cc:407] Connection header received
0927 21:15:10.723987 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.723988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.724000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.724045 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:10.724176 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.724646 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.725733 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.726109 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.726170 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.726182 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.726326 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.726367 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:15:10.726495 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.723754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52373 (local address 127.0.98.1:44231)
0927 21:15:10.723819 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.723821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.723829 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:15:10.723857 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.723858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.723866 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.723913 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:10.724052 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.724542 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.725777 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.726150 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.726216 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.726228 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.726384 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.726428 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:15:10.726495 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.723698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46619 (local address 127.0.98.3:48753)
0927 21:15:10.723858 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:10.723865 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:10.723925 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:10.724055 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:10.724055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:10.724056 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:10.724121 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.724123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.724724 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.724725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.725651 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.725654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.726188 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.726189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.726262 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.726266 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:15:10.726312 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:15:10.726355 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260927 21:15:10.726511 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.723649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44231 (local address 127.0.98.3:52373)
0927 21:15:10.723707 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:10.723718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:10.723763 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:10.723927 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:10.723928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:10.723930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:10.723994 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.723996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.724657 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.724658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.725685 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:10.725688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.726231 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.726233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:10.726316 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.726319 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:15:10.726365 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:15:10.726411 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260927 21:15:10.726891 15899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dadfbd08d104549842ac358a662860c" candidate_uuid: "66b910ab3ea94b90afd5aaac58169fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea660de4d814fad87e39ef67dca8521" is_pre_election: true
I20260927 21:15:10.726892 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dadfbd08d104549842ac358a662860c" candidate_uuid: "66b910ab3ea94b90afd5aaac58169fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58fbe5275644d39aa760b196c290609" is_pre_election: true
I20260927 21:15:10.727046 16033 raft_consensus.cc:2454] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b910ab3ea94b90afd5aaac58169fba in term 0.
I20260927 21:15:10.727047 15899 raft_consensus.cc:2454] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66b910ab3ea94b90afd5aaac58169fba in term 0.
I20260927 21:15:10.727294 16100 leader_election.cc:302] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5ea660de4d814fad87e39ef67dca8521, 66b910ab3ea94b90afd5aaac58169fba, a58fbe5275644d39aa760b196c290609; no voters:
I20260927 21:15:10.727420 16237 raft_consensus.cc:2781] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:15:10.727478 16237 raft_consensus.cc:482] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:15:10.727502 16237 raft_consensus.cc:3037] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:15:10.728199 16237 raft_consensus.cc:504] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:10.728336 16237 leader_election.cc:288] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [CANDIDATE]: Term 1 election: Requested vote from peers a58fbe5275644d39aa760b196c290609 (127.0.98.2:46619), 5ea660de4d814fad87e39ef67dca8521 (127.0.98.1:44231)
I20260927 21:15:10.728518 15899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dadfbd08d104549842ac358a662860c" candidate_uuid: "66b910ab3ea94b90afd5aaac58169fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea660de4d814fad87e39ef67dca8521"
I20260927 21:15:10.728595 15899 raft_consensus.cc:3037] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:15:10.729383 15899 raft_consensus.cc:2454] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b910ab3ea94b90afd5aaac58169fba in term 1.
I20260927 21:15:10.729372 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dadfbd08d104549842ac358a662860c" candidate_uuid: "66b910ab3ea94b90afd5aaac58169fba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58fbe5275644d39aa760b196c290609"
I20260927 21:15:10.729456 16033 raft_consensus.cc:3037] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:15:10.729589 16100 leader_election.cc:302] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea660de4d814fad87e39ef67dca8521, 66b910ab3ea94b90afd5aaac58169fba; no voters:
I20260927 21:15:10.729682 16237 raft_consensus.cc:2781] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:15:10.729839 16237 raft_consensus.cc:686] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 1 LEADER]: Becoming Leader. State: Replica: 66b910ab3ea94b90afd5aaac58169fba, State: Running, Role: LEADER
I20260927 21:15:10.729936 16237 consensus_queue.cc:237] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:10.730404 16033 raft_consensus.cc:2454] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66b910ab3ea94b90afd5aaac58169fba in term 1.
I20260927 21:15:10.730968 15753 catalog_manager.cc:5495] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba reported cstate change: term changed from 0 to 1, leader changed from <none> to 66b910ab3ea94b90afd5aaac58169fba (127.0.98.3). New cstate: current_term: 1 leader_uuid: "66b910ab3ea94b90afd5aaac58169fba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } health_report { overall_health: UNKNOWN } } }
I20260927 21:15:10.790342 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.787450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.0.98.3:36705)
0927 21:15:10.787503 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:10.787505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:10.787515 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:15:10.787573 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:10.787573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:10.787594 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:10.787660 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:15:10.787954 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.788446 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.789312 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:10.789633 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:10.789702 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:10.789936 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:15:10.790069 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:15:10.790143 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:10.790227 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 21:15:10.790252 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:15:10.792474 15899 raft_consensus.cc:1261] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 1 FOLLOWER]: Refusing update from remote peer 66b910ab3ea94b90afd5aaac58169fba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:15:10.792474 16033 raft_consensus.cc:1261] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [term 1 FOLLOWER]: Refusing update from remote peer 66b910ab3ea94b90afd5aaac58169fba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:15:10.792841 16237 consensus_queue.cc:1035] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:15:10.793015 16240 consensus_queue.cc:1035] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:15:10.797845 15754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:15:10.797978 15754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:15:10.798812 15754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Sending DeleteTablet for 3 replicas of tablet 1dadfbd08d104549842ac358a662860c
I20260927 21:15:10.799198 16147 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dadfbd08d104549842ac358a662860c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:15:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55704
I20260927 21:15:10.799214 15879 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dadfbd08d104549842ac358a662860c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:15:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54688
I20260927 21:15:10.799217 16013 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dadfbd08d104549842ac358a662860c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:15:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38376
I20260927 21:15:10.799527 16252 tablet_replica.cc:330] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: stopping tablet replica
I20260927 21:15:10.799546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 15816
I20260927 21:15:10.800400 16254 tablet_replica.cc:330] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: stopping tablet replica
I20260927 21:15:10.800532 16254 raft_consensus.cc:2229] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:15:10.800661 16254 raft_consensus.cc:2258] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:15:10.801102 16254 ts_tablet_manager.cc:1905] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:15:10.801561 16252 raft_consensus.cc:2229] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:15:10.801698 16252 raft_consensus.cc:2258] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:15:10.802255 16252 ts_tablet_manager.cc:1905] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:15:10.803148 16254 ts_tablet_manager.cc:1918] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:15:10.803390 16254 log.cc:1196] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/1dadfbd08d104549842ac358a662860c
I20260927 21:15:10.803805 16254 ts_tablet_manager.cc:1939] T 1dadfbd08d104549842ac358a662860c P a58fbe5275644d39aa760b196c290609: Deleting consensus metadata
I20260927 21:15:10.804322 16252 ts_tablet_manager.cc:1918] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:15:10.804430 16252 log.cc:1196] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/1dadfbd08d104549842ac358a662860c
I20260927 21:15:10.804411 15741 catalog_manager.cc:4841] TS a58fbe5275644d39aa760b196c290609 (127.0.98.2:46619): tablet 1dadfbd08d104549842ac358a662860c (table test-table [id=1dde208efa364fb795c659bb6eeb5cdd]) successfully deleted
I20260927 21:15:10.804689 16252 ts_tablet_manager.cc:1939] T 1dadfbd08d104549842ac358a662860c P 66b910ab3ea94b90afd5aaac58169fba: Deleting consensus metadata
I20260927 21:15:10.805080 15739 catalog_manager.cc:4841] TS 66b910ab3ea94b90afd5aaac58169fba (127.0.98.3:36705): tablet 1dadfbd08d104549842ac358a662860c (table test-table [id=1dde208efa364fb795c659bb6eeb5cdd]) successfully deleted
W20260927 21:15:10.805905 15738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44231 (error 108)
I20260927 21:15:10.806457 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:10.806335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44231 (local address 127.0.0.1:52544)
0927 21:15:10.806388 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:10.806424 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:15:10.806470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 15950
W20260927 21:15:10.806572 15738 catalog_manager.cc:4584] TS 5ea660de4d814fad87e39ef67dca8521 (127.0.98.1:44231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1dadfbd08d104549842ac358a662860c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44231: connect: Connection refused (error 111)
I20260927 21:15:10.812455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 16084
I20260927 21:15:10.817436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 15722
I20260927 21:15:30.822925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46331
--webserver_interface=127.0.98.62
--webserver_port=34247
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:15:30.904867 16256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:30.905122 16256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:30.905146 16256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:30.906474 16256 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:15:30.906534 16256 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:15:30.906546 16256 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:15:30.906558 16256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:15:30.906569 16256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:30.906580 16256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:30.906591 16256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:15:30.906605 16256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:15:30.908259 16256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:30.908515 16256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:30.908757 16256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:15:30.920702 16256 server_base.cc:1034] running on GCE node
W20260927 21:15:30.920661 16262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:30.920665 16265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:30.920655 16263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:30.921176 16256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:30.921543 16256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:30.922722 16256 hybrid_clock.cc:648] HybridClock initialized: now 1790543730922707 us; error 51 us; skew 500 ppm
Sep 27 21:15:30 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:30.929546 16256 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:15:30.930063 16256 webserver.cc:460] Webserver started at http://127.0.98.62:34247/ using document root <none> and password file <none>
I20260927 21:15:30.930290 16256 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:30.930356 16256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:30.932682 16256 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:15:30.933538 16272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:30.934041 16256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:30.934176 16256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "ab41038cbe4648159c32b0c6b7c130dd"
format_stamp: "Formatted at 2026-09-27 21:15:09 on dist-test-slave-r9x1"
server_key: "75d241e5d621ad68d8465c4f26c9ef4e"
server_key_iv: "4c0c551ac27d9494c34e6ee7c377b6cc"
server_key_version: "encryptionkey@0"
I20260927 21:15:30.934670 16256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:30.955651 16256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:30.957677 16256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:30.957815 16256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:30.961635 16256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46331
I20260927 21:15:30.962000 16324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46331 every 8 connection(s)
I20260927 21:15:30.962051 16256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:15:30.963378 16325 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:15:30.964054 16325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Bootstrap starting.
I20260927 21:15:30.968597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 16256
I20260927 21:15:30.968876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44231
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46331
--webserver_port=36897
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:15:30.973529 16325 log.cc:824] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Log is configured to *not* fsync() on all Append() calls
I20260927 21:15:30.975416 16325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:15:30.975692 16325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Bootstrap complete.
I20260927 21:15:30.977877 16325 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:30.978125 16325 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab41038cbe4648159c32b0c6b7c130dd, State: Initialized, Role: FOLLOWER
I20260927 21:15:30.978278 16325 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:30.978353 16325 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:15:30.978406 16325 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:15:30.978456 16325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:15:30.979523 16325 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:30.979624 16325 leader_election.cc:302] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab41038cbe4648159c32b0c6b7c130dd; no voters:
I20260927 21:15:30.979792 16325 leader_election.cc:288] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:15:30.979888 16329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:15:30.980075 16325 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:15:30.980082 16329 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 2 LEADER]: Becoming Leader. State: Replica: ab41038cbe4648159c32b0c6b7c130dd, State: Running, Role: LEADER
I20260927 21:15:30.980255 16329 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:30.980587 16330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ab41038cbe4648159c32b0c6b7c130dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } } }
I20260927 21:15:30.980667 16330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: This master's current role is: LEADER
I20260927 21:15:30.981014 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab41038cbe4648159c32b0c6b7c130dd. Latest consensus state: current_term: 2 leader_uuid: "ab41038cbe4648159c32b0c6b7c130dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } } }
I20260927 21:15:30.981094 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: This master's current role is: LEADER
I20260927 21:15:30.981208 16337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:15:30.982064 16337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:15:30.982159 16337 catalog_manager.cc:1259] Loaded cluster ID: cc2256f79cbc4c18a37f403902859a04
I20260927 21:15:30.982190 16337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:15:30.983031 16337 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:15:30.983211 16337 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Loaded TSK: 0
I20260927 21:15:30.995239 16337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Generated new TSK 1
I20260927 21:15:30.995352 16337 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:15:31.080132 16327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:31.080397 16327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:31.080435 16327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:31.082069 16327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:31.082140 16327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:31.082186 16327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:15:31.083961 16327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36897
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:31.084235 16327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:31.084462 16327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:15:31.090330 16327 server_base.cc:1034] running on GCE node
W20260927 21:15:31.090446 16355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:31.090325 16353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:31.090636 16352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:31.090895 16327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:31.091178 16327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:31.092376 16327 hybrid_clock.cc:648] HybridClock initialized: now 1790543731092355 us; error 36 us; skew 500 ppm
Sep 27 21:15:31 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:31.096356 16327 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:15:31.097036 16327 webserver.cc:460] Webserver started at http://127.0.98.1:36897/ using document root <none> and password file <none>
I20260927 21:15:31.097364 16327 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:31.097445 16327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:31.099460 16327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:15:31.100445 16362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:31.100705 16327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:31.100816 16327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "5ea660de4d814fad87e39ef67dca8521"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "d774d30e4a018334e3906976c98dc59b"
server_key_iv: "76a59784715e468c2043874b994ae589"
server_key_version: "encryptionkey@0"
I20260927 21:15:31.101243 16327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:31.120473 16327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:31.123665 16327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:31.123834 16327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:31.124083 16327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:15:31.124528 16369 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:15:31.126968 16327 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:15:31.127048 16327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260927 21:15:31.127089 16327 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:15:31.127769 16327 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:15:31.127805 16327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:15:31.127923 16369 tablet_bootstrap.cc:492] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Bootstrap starting.
I20260927 21:15:31.136042 16327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44231
I20260927 21:15:31.136125 16476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44231 every 8 connection(s)
I20260927 21:15:31.136559 16327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:15:31.140477 16369 log.cc:824] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Log is configured to *not* fsync() on all Append() calls
I20260927 21:15:31.143246 16369 tablet_bootstrap.cc:492] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:15:31.143548 16369 tablet_bootstrap.cc:492] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Bootstrap complete.
I20260927 21:15:31.144282 16369 ts_tablet_manager.cc:1397] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20260927 21:15:31.145737 16369 raft_consensus.cc:348] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.146057 16369 raft_consensus.cc:729] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea660de4d814fad87e39ef67dca8521, State: Initialized, Role: FOLLOWER
I20260927 21:15:31.146237 16369 consensus_queue.cc:260] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.146516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 16327
I20260927 21:15:31.146565 16369 ts_tablet_manager.cc:1428] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260927 21:15:31.146783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46619
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46331
--webserver_port=38683
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:15:31 dist-test-slave-r9x1 krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:15:31.153398 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.137310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:57733)
0927 21:15:31.140823 (+ 3513us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:31.140883 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:31.141612 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:31.142137 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:31.142142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:31.142507 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:15:31.142802 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.142809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.144525 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.144527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.145836 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.145839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.146829 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.146831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.146982 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.147600 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:15:31.147610 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:15:31.149498 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:15:31.151730 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:31.151742 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:31.151744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:31.151991 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:31.152244 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:31.152245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:31.152245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:31.152359 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:31.152725 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:15:31.152731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:15:31.152867 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:15:31.152950 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:15:31.153148 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3455,"thread_start_us":93,"threads_started":1}
I20260927 21:15:31.153396 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.137444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57733 (local address 127.0.98.62:46331)
0927 21:15:31.137652 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.137662 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.141080 (+ 3418us) server_negotiation.cc:407] Connection header received
0927 21:15:31.141877 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.141886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.141947 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.142089 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:31.142948 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.144396 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.145950 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.146604 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.146720 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.149631 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:31.149637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:31.149637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:31.149652 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:31.151600 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.152086 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.152089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.152089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.152181 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.152427 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.152428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.152429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.152660 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:31.152731 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.153039 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.153171 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260927 21:15:31.154134 16477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46331
I20260927 21:15:31.154240 16477 heartbeater.cc:464] Registering TS with master...
I20260927 21:15:31.154434 16477 heartbeater.cc:511] Master 127.0.98.62:46331 requested a full tablet report, sending...
I20260927 21:15:31.155488 16289 ts_manager.cc:194] Registered new tserver with Master: 5ea660de4d814fad87e39ef67dca8521 (127.0.98.1:44231)
I20260927 21:15:31.156659 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57733
I20260927 21:15:31.157639 16477 heartbeater.cc:503] Master 127.0.98.62:46331 was elected leader, sending a full tablet report...
I20260927 21:15:31.162026 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.156175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44231 (local address 127.0.0.1:44216)
0927 21:15:31.157380 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:31.157429 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:31.157512 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:31.157861 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:31.157865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:31.157868 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:31.157942 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.157944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.159390 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.159392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.160627 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.160634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.161709 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.161710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.161803 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.161817 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:15:31.161891 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:15:31.161954 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":103,"threads_started":1}
I20260927 21:15:31.162050 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.156456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.0.98.1:44231)
0927 21:15:31.157666 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.157671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.157682 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:15:31.157720 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.157723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.157740 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.157832 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:31.158032 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.159160 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.160746 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.161584 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.161672 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.161735 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.161919 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.161965 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1171,"thread_start_us":72,"threads_started":1}
I20260927 21:15:31.162511 16411 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dadfbd08d104549842ac358a662860c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:15:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44216
I20260927 21:15:31.165472 16486 tablet_replica.cc:330] stopping tablet replica
I20260927 21:15:31.165637 16486 raft_consensus.cc:2229] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:15:31.165705 16486 raft_consensus.cc:2258] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:15:31.166085 16486 ts_tablet_manager.cc:1905] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:15:31.168671 16486 ts_tablet_manager.cc:1918] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:15:31.168781 16486 log.cc:1196] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/1dadfbd08d104549842ac358a662860c
I20260927 21:15:31.168999 16486 ts_tablet_manager.cc:1939] T 1dadfbd08d104549842ac358a662860c P 5ea660de4d814fad87e39ef67dca8521: Deleting consensus metadata
I20260927 21:15:31.169914 16273 catalog_manager.cc:4841] TS 5ea660de4d814fad87e39ef67dca8521 (127.0.98.1:44231): tablet 1dadfbd08d104549842ac358a662860c (table test-table [id=1dde208efa364fb795c659bb6eeb5cdd]) successfully deleted
W20260927 21:15:31.235742 16483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:31.235973 16483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:31.235999 16483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:31.237252 16483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:31.237329 16483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:31.237358 16483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:15:31.239003 16483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38683
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:31.239253 16483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:31.239488 16483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:15:31.244537 16494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:31.244624 16483 server_base.cc:1034] running on GCE node
W20260927 21:15:31.244571 16491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:31.244711 16492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:31.244941 16483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:31.245165 16483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:31.246321 16483 hybrid_clock.cc:648] HybridClock initialized: now 1790543731246313 us; error 40 us; skew 500 ppm
Sep 27 21:15:31 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:31.249478 16483 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:15:31.250056 16483 webserver.cc:460] Webserver started at http://127.0.98.2:38683/ using document root <none> and password file <none>
I20260927 21:15:31.250293 16483 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:31.250370 16483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:31.251995 16483 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:15:31.252696 16501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:31.252889 16483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:31.252974 16483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "a58fbe5275644d39aa760b196c290609"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "c9ccdc55a1bdd728426bca2365244613"
server_key_iv: "42254ee8e47220a40e1ed43d8754278f"
server_key_version: "encryptionkey@0"
I20260927 21:15:31.253376 16483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:31.267138 16483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:31.269518 16483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:31.269670 16483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:31.269912 16483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:15:31.270226 16483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:15:31.270262 16483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:31.270292 16483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:15:31.270310 16483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:31.276818 16483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46619
I20260927 21:15:31.276866 16614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46619 every 8 connection(s)
I20260927 21:15:31.277216 16483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:15:31.283008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 16483
I20260927 21:15:31.283313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36705
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46331
--webserver_port=43949
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:15:31 dist-test-slave-r9x1 krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:15:31.288395 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.277981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52937 (local address 127.0.98.62:46331)
0927 21:15:31.278068 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.278070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.278225 (+ 155us) server_negotiation.cc:407] Connection header received
0927 21:15:31.278780 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.278781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.278793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.278861 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:31.279909 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.280477 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.281901 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.282179 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.282287 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.285181 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:31.285183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:31.285183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:31.285192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:31.286649 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.287203 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.287204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.287204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.287267 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.287609 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.287610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.287610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.287747 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:31.287813 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.288191 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.288320 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:15:31.288597 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.277956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:52937)
0927 21:15:31.278107 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:31.278144 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:31.278659 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:31.278927 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:31.278931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:31.279537 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:15:31.279800 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.279806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.280738 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.280739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.281731 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.281734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.282299 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.282300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.282413 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.283249 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:15:31.283274 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:15:31.285035 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:15:31.286742 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:31.286751 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:31.286753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:31.287101 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:31.287369 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:31.287369 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:31.287370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:31.287514 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:31.287818 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:15:31.287823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:15:31.288020 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 21:15:31.288144 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 21:15:31.288339 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260927 21:15:31.289474 16615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46331
I20260927 21:15:31.289577 16615 heartbeater.cc:464] Registering TS with master...
I20260927 21:15:31.289763 16615 heartbeater.cc:511] Master 127.0.98.62:46331 requested a full tablet report, sending...
I20260927 21:15:31.290329 16289 ts_manager.cc:194] Registered new tserver with Master: a58fbe5275644d39aa760b196c290609 (127.0.98.2:46619)
I20260927 21:15:31.290884 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52937
W20260927 21:15:31.366765 16618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:31.366994 16618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:31.367020 16618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:31.368289 16618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:31.368345 16618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:31.368374 16618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:15:31.369989 16618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43949
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:31.370247 16618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:31.370486 16618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:15:31.375921 16624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:31.375924 16626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:31.375921 16623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:31.376535 16618 server_base.cc:1034] running on GCE node
I20260927 21:15:31.376770 16618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:31.377043 16618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:31.378207 16618 hybrid_clock.cc:648] HybridClock initialized: now 1790543731378172 us; error 49 us; skew 500 ppm
Sep 27 21:15:31 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:31.380874 16618 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:15:31.381488 16618 webserver.cc:460] Webserver started at http://127.0.98.3:43949/ using document root <none> and password file <none>
I20260927 21:15:31.381723 16618 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:31.381788 16618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:31.383594 16618 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:15:31.384214 16633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:31.384366 16618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:15:31.384459 16618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "66b910ab3ea94b90afd5aaac58169fba"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "a74488d3119bd6ba5737dfd698e92dd3"
server_key_iv: "62f2aab0ac0573ab5653a47d368bc8fb"
server_key_version: "encryptionkey@0"
I20260927 21:15:31.384864 16618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:31.407321 16618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:31.409772 16618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:31.409929 16618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:31.410184 16618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:15:31.410485 16618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:15:31.410522 16618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:31.410552 16618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:15:31.410578 16618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:31.417078 16618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36705
I20260927 21:15:31.417126 16746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36705 every 8 connection(s)
I20260927 21:15:31.417528 16618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:15:31.419653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 16618
Sep 27 21:15:31 dist-test-slave-r9x1 krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:15:31.427857 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.420060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.0.98.62:46331)
0927 21:15:31.420160 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.420162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.420246 (+ 84us) server_negotiation.cc:407] Connection header received
0927 21:15:31.420324 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.420324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.420337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.420414 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:31.421121 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.421625 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.422859 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.423048 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.423096 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.424357 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:31.424358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:31.424359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:31.424367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:31.426543 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.427089 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.427090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.427091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.427153 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.427329 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.427330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.427330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.427451 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:31.427632 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.427670 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.427758 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":47,"threads_started":1}
I20260927 21:15:31.428180 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.418319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53003 (local address 127.0.98.62:46331)
0927 21:15:31.418403 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.418405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.418582 (+ 177us) server_negotiation.cc:407] Connection header received
0927 21:15:31.419142 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.419144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.419157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.419210 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:31.420307 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.420847 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.422517 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.422714 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.422776 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.425323 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:31.425326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:31.425326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:31.425334 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:31.426824 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.427246 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.427247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.427247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.427302 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.427531 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.427532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.427532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.427641 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:31.427695 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.428034 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.428106 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:15:31.428419 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.418293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:53003)
0927 21:15:31.418437 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:31.418483 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:31.419026 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:31.419287 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:31.419291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:31.419931 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:15:31.420222 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.420230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.420956 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.420957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.422400 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.422405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.422799 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.422800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.422946 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.423467 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:15:31.423478 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:15:31.425214 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:15:31.426916 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:31.426925 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:31.426928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:31.427175 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:31.427357 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:15:31.427358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:15:31.427358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:15:31.427476 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:15:31.427696 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:15:31.427702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:15:31.427857 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:15:31.427997 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 21:15:31.428182 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260927 21:15:31.428892 16747 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46331
I20260927 21:15:31.429037 16747 heartbeater.cc:464] Registering TS with master...
I20260927 21:15:31.429234 16747 heartbeater.cc:511] Master 127.0.98.62:46331 requested a full tablet report, sending...
I20260927 21:15:31.429924 16289 ts_manager.cc:194] Registered new tserver with Master: 66b910ab3ea94b90afd5aaac58169fba (127.0.98.3:36705)
I20260927 21:15:31.430454 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53003
I20260927 21:15:31.440165 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:15:31.443130 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.440640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.0.98.62:46331)
0927 21:15:31.440706 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.440708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.440800 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:15:31.440895 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.440895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.440943 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.441011 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:15:31.441137 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.441676 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.442441 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.442643 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.442697 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.442847 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:15:31.442901 (+ 54us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 21:15:31.443054 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55310: token expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 21:15:31.443197 16480 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55310: token expired
W20260927 21:15:31.443331 16217 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46331)
I20260927 21:15:31.448982 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.443602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.0.98.62:46331)
0927 21:15:31.443659 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.443661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.443669 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:15:31.443687 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.443688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.443696 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.443747 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:15:31.444237 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.444693 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.445428 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.445611 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.445663 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.446757 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:15:31.446758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:15:31.446759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:15:31.446766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:15:31.447917 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.448212 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.448213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.448214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.448266 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:15:31.448429 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:15:31.448430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:15:31.448430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:15:31.448588 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:15:31.448679 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.448796 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.448888 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:15:31.450259 16289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:15:31.451916 16289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:15:31.454454 16411 tablet_service.cc:1467] Processing CreateTablet for tablet d516920a09d5420d8a3f8d269eff0405 (DEFAULT_TABLE table=test-table [id=3f9c8b366d4949a49fe112be49993f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:15:31.454672 16411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d516920a09d5420d8a3f8d269eff0405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:15:31.456678 16369 tablet_bootstrap.cc:492] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: Bootstrap starting.
I20260927 21:15:31.457011 16369 tablet_bootstrap.cc:654] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: Neither blocks nor log segments found. Creating new log.
I20260927 21:15:31.457631 16369 tablet_bootstrap.cc:492] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: No bootstrap required, opened a new log
I20260927 21:15:31.457692 16369 ts_tablet_manager.cc:1397] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:31.457803 16369 raft_consensus.cc:348] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.457903 16369 raft_consensus.cc:374] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:15:31.457959 16369 raft_consensus.cc:729] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea660de4d814fad87e39ef67dca8521, State: Initialized, Role: FOLLOWER
I20260927 21:15:31.458009 16369 consensus_queue.cc:260] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.458110 16369 ts_tablet_manager.cc:1428] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:15:31.458182 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.454176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46619 (local address 127.0.0.1:36290)
0927 21:15:31.454305 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:31.454312 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:31.454345 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:31.454756 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:31.454757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:31.454759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:31.454821 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.454823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.455783 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.455784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.456696 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.456698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.457905 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.457906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.457993 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.457998 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:15:31.458052 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:15:31.458101 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260927 21:15:31.458451 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.454401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.0.98.2:46619)
0927 21:15:31.454589 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.454593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.454604 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:15:31.454646 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.454648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.454663 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.454750 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:31.454876 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.455709 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.457041 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.457826 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.457905 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.457978 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.458070 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.458119 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260927 21:15:31.459539 16549 tablet_service.cc:1467] Processing CreateTablet for tablet d516920a09d5420d8a3f8d269eff0405 (DEFAULT_TABLE table=test-table [id=3f9c8b366d4949a49fe112be49993f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:15:31.459941 16549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d516920a09d5420d8a3f8d269eff0405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:15:31.460019 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.454120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.0.1:60400)
0927 21:15:31.454171 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:31.454182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:31.454219 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:31.454945 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:31.454947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:31.454952 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:31.455141 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.455145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.456523 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.456524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.457521 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.457524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.459761 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.459762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.459835 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.459840 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:15:31.459893 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:15:31.459950 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:15:31.460348 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.454399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.0.98.3:36705)
0927 21:15:31.454583 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.454588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.454599 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:15:31.454640 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.454642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.454657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.455285 (+ 628us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:31.455390 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.456422 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.457870 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.459493 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.459586 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.459650 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.460217 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.460270 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260927 21:15:31.461030 16681 tablet_service.cc:1467] Processing CreateTablet for tablet d516920a09d5420d8a3f8d269eff0405 (DEFAULT_TABLE table=test-table [id=3f9c8b366d4949a49fe112be49993f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:15:31.461361 16681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d516920a09d5420d8a3f8d269eff0405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:15:31.463047 16757 tablet_bootstrap.cc:492] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: Bootstrap starting.
I20260927 21:15:31.463649 16757 tablet_bootstrap.cc:654] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: Neither blocks nor log segments found. Creating new log.
I20260927 21:15:31.464244 16757 log.cc:824] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: Log is configured to *not* fsync() on all Append() calls
I20260927 21:15:31.464257 16758 tablet_bootstrap.cc:492] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: Bootstrap starting.
I20260927 21:15:31.464879 16758 tablet_bootstrap.cc:654] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: Neither blocks nor log segments found. Creating new log.
I20260927 21:15:31.465040 16757 tablet_bootstrap.cc:492] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: No bootstrap required, opened a new log
I20260927 21:15:31.465122 16757 ts_tablet_manager.cc:1397] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:15:31.465512 16758 log.cc:824] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: Log is configured to *not* fsync() on all Append() calls
I20260927 21:15:31.466365 16758 tablet_bootstrap.cc:492] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: No bootstrap required, opened a new log
I20260927 21:15:31.466445 16758 ts_tablet_manager.cc:1397] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:15:31.467157 16757 raft_consensus.cc:348] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.467376 16757 raft_consensus.cc:374] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:15:31.467414 16757 raft_consensus.cc:729] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58fbe5275644d39aa760b196c290609, State: Initialized, Role: FOLLOWER
I20260927 21:15:31.467551 16757 consensus_queue.cc:260] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.467850 16757 ts_tablet_manager.cc:1428] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:15:31.467885 16615 heartbeater.cc:503] Master 127.0.98.62:46331 was elected leader, sending a full tablet report...
I20260927 21:15:31.468380 16758 raft_consensus.cc:348] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.468526 16758 raft_consensus.cc:374] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:15:31.468570 16758 raft_consensus.cc:729] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b910ab3ea94b90afd5aaac58169fba, State: Initialized, Role: FOLLOWER
I20260927 21:15:31.468689 16758 consensus_queue.cc:260] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.468990 16758 ts_tablet_manager.cc:1428] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:15:31.468990 16747 heartbeater.cc:503] Master 127.0.98.62:46331 was elected leader, sending a full tablet report...
W20260927 21:15:31.527853 16616 tablet.cc:2369] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:15:31.545920 16482 raft_consensus.cc:482] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:15:31.546052 16482 raft_consensus.cc:504] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.546401 16482 leader_election.cc:288] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66b910ab3ea94b90afd5aaac58169fba (127.0.98.3:36705), a58fbe5275644d39aa760b196c290609 (127.0.98.2:46619)
I20260927 21:15:31.549640 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.546631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46619 (local address 127.0.98.1:56257)
0927 21:15:31.546768 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:31.546787 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:31.546891 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:31.547164 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:31.547166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:31.547169 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:31.547248 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.547250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.547953 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.547955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.548776 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.548779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.549390 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.549391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.549476 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.549480 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:15:31.549533 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:15:31.549573 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 21:15:31.549818 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.546863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56257 (local address 127.0.98.2:46619)
0927 21:15:31.546964 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.546967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.546980 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:15:31.547011 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.547013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.547027 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.547090 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:31.547319 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.547869 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.548883 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.549315 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.549371 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.549382 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.549631 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.549722 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:15:31.550103 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.546631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.98.1:53741)
0927 21:15:31.546898 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:31.546910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:31.547004 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:31.547274 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:15:31.547275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:15:31.547276 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:15:31.547409 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.547412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.548051 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.548051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.548838 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:15:31.548841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.549624 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.549627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:15:31.549744 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.549752 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:15:31.549911 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 21:15:31.549974 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":94,"thread_start_us":103,"threads_started":1}
I20260927 21:15:31.550200 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.546774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53741 (local address 127.0.98.3:36705)
0927 21:15:31.546825 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.546827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.547008 (+ 181us) server_negotiation.cc:407] Connection header received
0927 21:15:31.547165 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.547166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.547182 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.547267 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:15:31.547492 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.547971 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.548933 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.549388 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.549460 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.549471 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.550072 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.550129 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:15:31.550225 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516920a09d5420d8a3f8d269eff0405" candidate_uuid: "5ea660de4d814fad87e39ef67dca8521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58fbe5275644d39aa760b196c290609" is_pre_election: true
I20260927 21:15:31.550413 16569 raft_consensus.cc:2454] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea660de4d814fad87e39ef67dca8521 in term 0.
I20260927 21:15:31.550544 16701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516920a09d5420d8a3f8d269eff0405" candidate_uuid: "5ea660de4d814fad87e39ef67dca8521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b910ab3ea94b90afd5aaac58169fba" is_pre_election: true
I20260927 21:15:31.550680 16701 raft_consensus.cc:2454] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ea660de4d814fad87e39ef67dca8521 in term 0.
I20260927 21:15:31.550730 16366 leader_election.cc:302] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea660de4d814fad87e39ef67dca8521, a58fbe5275644d39aa760b196c290609; no voters:
I20260927 21:15:31.550933 16482 raft_consensus.cc:2781] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:15:31.551007 16482 raft_consensus.cc:482] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:15:31.551030 16482 raft_consensus.cc:3037] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:15:31.551725 16482 raft_consensus.cc:504] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.551868 16482 leader_election.cc:288] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [CANDIDATE]: Term 1 election: Requested vote from peers 66b910ab3ea94b90afd5aaac58169fba (127.0.98.3:36705), a58fbe5275644d39aa760b196c290609 (127.0.98.2:46619)
I20260927 21:15:31.552071 16701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516920a09d5420d8a3f8d269eff0405" candidate_uuid: "5ea660de4d814fad87e39ef67dca8521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b910ab3ea94b90afd5aaac58169fba"
I20260927 21:15:31.552069 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d516920a09d5420d8a3f8d269eff0405" candidate_uuid: "5ea660de4d814fad87e39ef67dca8521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a58fbe5275644d39aa760b196c290609"
I20260927 21:15:31.552172 16701 raft_consensus.cc:3037] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:15:31.552191 16569 raft_consensus.cc:3037] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:15:31.553078 16569 raft_consensus.cc:2454] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea660de4d814fad87e39ef67dca8521 in term 1.
I20260927 21:15:31.553078 16701 raft_consensus.cc:2454] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ea660de4d814fad87e39ef67dca8521 in term 1.
I20260927 21:15:31.553288 16364 leader_election.cc:302] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea660de4d814fad87e39ef67dca8521, 66b910ab3ea94b90afd5aaac58169fba; no voters:
I20260927 21:15:31.553407 16482 raft_consensus.cc:2781] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:15:31.553544 16482 raft_consensus.cc:686] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 1 LEADER]: Becoming Leader. State: Replica: 5ea660de4d814fad87e39ef67dca8521, State: Running, Role: LEADER
I20260927 21:15:31.553602 16482 consensus_queue.cc:237] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } }
I20260927 21:15:31.554332 16289 catalog_manager.cc:5495] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ea660de4d814fad87e39ef67dca8521 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5ea660de4d814fad87e39ef67dca8521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ea660de4d814fad87e39ef67dca8521" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44231 } health_report { overall_health: HEALTHY } } }
I20260927 21:15:31.580958 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.578361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.0.98.1:44231)
0927 21:15:31.578423 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:15:31.578425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:15:31.578434 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:15:31.578476 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:15:31.578477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:15:31.578510 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:15:31.578571 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:15:31.578688 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.579183 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.580077 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:15:31.580279 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:15:31.580331 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:15:31.580493 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:15:31.580620 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:15:31.580685 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:15:31.580807 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 21:15:31.580838 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:15:31.582880 16701 raft_consensus.cc:1261] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [term 1 FOLLOWER]: Refusing update from remote peer 5ea660de4d814fad87e39ef67dca8521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:15:31.582888 16569 raft_consensus.cc:1261] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [term 1 FOLLOWER]: Refusing update from remote peer 5ea660de4d814fad87e39ef67dca8521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:15:31.583375 16764 consensus_queue.cc:1035] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b910ab3ea94b90afd5aaac58169fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:15:31.583701 16482 consensus_queue.cc:1035] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a58fbe5275644d39aa760b196c290609" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:15:31.587561 16768 mvcc.cc:205] Tried to move back new op lower bound from 7334067124560793600 to 7334067124444094464. Current Snapshot: MvccSnapshot[applied={T|T < 7334067124560793600}]
I20260927 21:15:31.587723 16769 mvcc.cc:205] Tried to move back new op lower bound from 7334067124560793600 to 7334067124444094464. Current Snapshot: MvccSnapshot[applied={T|T < 7334067124560793600 or (T in {7334067124560793600})}]
I20260927 21:15:31.589684 16288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:15:31.589799 16288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:15:31.590706 16288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Sending DeleteTablet for 3 replicas of tablet d516920a09d5420d8a3f8d269eff0405
I20260927 21:15:31.591027 16411 tablet_service.cc:1514] Processing DeleteTablet for tablet d516920a09d5420d8a3f8d269eff0405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:15:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44216
I20260927 21:15:31.591080 16549 tablet_service.cc:1514] Processing DeleteTablet for tablet d516920a09d5420d8a3f8d269eff0405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:15:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36290
I20260927 21:15:31.591177 16681 tablet_service.cc:1514] Processing DeleteTablet for tablet d516920a09d5420d8a3f8d269eff0405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:15:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60400
I20260927 21:15:31.591413 16486 tablet_replica.cc:330] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: stopping tablet replica
I20260927 21:15:31.591473 16777 tablet_replica.cc:330] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: stopping tablet replica
I20260927 21:15:31.591480 16486 raft_consensus.cc:2229] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:15:31.591576 16486 raft_consensus.cc:2258] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:15:31.591596 16777 raft_consensus.cc:2229] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:15:31.591704 16777 raft_consensus.cc:2258] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:15:31.591966 16486 ts_tablet_manager.cc:1905] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:15:31.592180 16777 ts_tablet_manager.cc:1905] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:15:31.593441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 16327
I20260927 21:15:31.594261 16777 ts_tablet_manager.cc:1918] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:15:31.594347 16777 log.cc:1196] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/d516920a09d5420d8a3f8d269eff0405
I20260927 21:15:31.594571 16777 ts_tablet_manager.cc:1939] T d516920a09d5420d8a3f8d269eff0405 P a58fbe5275644d39aa760b196c290609: Deleting consensus metadata
I20260927 21:15:31.594566 16778 tablet_replica.cc:330] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: stopping tablet replica
I20260927 21:15:31.594694 16778 raft_consensus.cc:2229] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:15:31.594851 16778 raft_consensus.cc:2258] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:15:31.595119 16276 catalog_manager.cc:4841] TS a58fbe5275644d39aa760b196c290609 (127.0.98.2:46619): tablet d516920a09d5420d8a3f8d269eff0405 (table test-table [id=3f9c8b366d4949a49fe112be49993f46]) successfully deleted
I20260927 21:15:31.595342 16778 ts_tablet_manager.cc:1905] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:15:31.597061 16778 ts_tablet_manager.cc:1918] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:15:31.597150 16778 log.cc:1196] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/d516920a09d5420d8a3f8d269eff0405
I20260927 21:15:31.597457 16778 ts_tablet_manager.cc:1939] T d516920a09d5420d8a3f8d269eff0405 P 66b910ab3ea94b90afd5aaac58169fba: Deleting consensus metadata
I20260927 21:15:31.598014 16274 catalog_manager.cc:4841] TS 66b910ab3ea94b90afd5aaac58169fba (127.0.98.3:36705): tablet d516920a09d5420d8a3f8d269eff0405 (table test-table [id=3f9c8b366d4949a49fe112be49993f46]) successfully deleted
W20260927 21:15:31.600035 16273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44231 (error 108)
I20260927 21:15:31.600427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 16483
I20260927 21:15:31.601158 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:31.600550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44231 (local address 127.0.0.1:44234)
0927 21:15:31.601066 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:31.601120 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492}
W20260927 21:15:31.602646 16273 catalog_manager.cc:4584] TS 5ea660de4d814fad87e39ef67dca8521 (127.0.98.1:44231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d516920a09d5420d8a3f8d269eff0405: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44231: connect: Connection refused (error 111)
I20260927 21:15:31.605737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 16618
I20260927 21:15:31.610159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 16256
I20260927 21:15:51.615527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46331
--webserver_interface=127.0.98.62
--webserver_port=34247
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:15:51.699230 16780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:51.699469 16780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:51.699493 16780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:51.700834 16780 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:15:51.700888 16780 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:15:51.700901 16780 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:15:51.700910 16780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:15:51.700922 16780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:51.700934 16780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:51.700946 16780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:15:51.700961 16780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:15:51.702688 16780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:51.702965 16780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:51.703240 16780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:15:51.708195 16786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:51.708410 16780 server_base.cc:1034] running on GCE node
W20260927 21:15:51.708184 16788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:51.708182 16785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:51.708742 16780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:51.709048 16780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:51.710250 16780 hybrid_clock.cc:648] HybridClock initialized: now 1790543751710198 us; error 72 us; skew 500 ppm
Sep 27 21:15:51 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:51.713198 16780 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:15:51.713716 16780 webserver.cc:460] Webserver started at http://127.0.98.62:34247/ using document root <none> and password file <none>
I20260927 21:15:51.713953 16780 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:51.714005 16780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:51.715396 16780 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:15:51.716080 16795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:51.716260 16780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:15:51.716343 16780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "ab41038cbe4648159c32b0c6b7c130dd"
format_stamp: "Formatted at 2026-09-27 21:15:09 on dist-test-slave-r9x1"
server_key: "75d241e5d621ad68d8465c4f26c9ef4e"
server_key_iv: "4c0c551ac27d9494c34e6ee7c377b6cc"
server_key_version: "encryptionkey@0"
I20260927 21:15:51.716699 16780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:51.726462 16780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:51.728694 16780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:51.728842 16780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:51.732815 16780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46331
I20260927 21:15:51.732851 16847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46331 every 8 connection(s)
I20260927 21:15:51.733245 16780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:15:51.734808 16848 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:15:51.735491 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Bootstrap starting.
I20260927 21:15:51.741019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 16780
I20260927 21:15:51.741333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44231
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46331
--webserver_port=36897
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:15:51.744068 16848 log.cc:824] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Log is configured to *not* fsync() on all Append() calls
I20260927 21:15:51.746367 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:15:51.746670 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Bootstrap complete.
I20260927 21:15:51.748782 16848 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:51.749027 16848 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab41038cbe4648159c32b0c6b7c130dd, State: Initialized, Role: FOLLOWER
I20260927 21:15:51.749197 16848 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:51.749301 16848 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:15:51.749356 16848 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:15:51.749404 16848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:15:51.750461 16848 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:51.750567 16848 leader_election.cc:302] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab41038cbe4648159c32b0c6b7c130dd; no voters:
I20260927 21:15:51.750739 16848 leader_election.cc:288] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:15:51.750788 16852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:15:51.750942 16852 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [term 3 LEADER]: Becoming Leader. State: Replica: ab41038cbe4648159c32b0c6b7c130dd, State: Running, Role: LEADER
I20260927 21:15:51.751019 16848 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:15:51.751092 16852 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 21:15:51.751479 16853 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ab41038cbe4648159c32b0c6b7c130dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } } }
I20260927 21:15:51.751564 16853 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: This master's current role is: LEADER
I20260927 21:15:51.751801 16854 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab41038cbe4648159c32b0c6b7c130dd. Latest consensus state: current_term: 3 leader_uuid: "ab41038cbe4648159c32b0c6b7c130dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab41038cbe4648159c32b0c6b7c130dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } } }
I20260927 21:15:51.751909 16854 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd [sys.catalog]: This master's current role is: LEADER
I20260927 21:15:51.751965 16859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:15:51.752871 16859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:15:51.753070 16859 catalog_manager.cc:1259] Loaded cluster ID: cc2256f79cbc4c18a37f403902859a04
I20260927 21:15:51.753324 16859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:15:51.754536 16859 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:15:51.754757 16859 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Loaded TSK: 0
I20260927 21:15:51.754809 16859 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ab41038cbe4648159c32b0c6b7c130dd: Loaded TSK: 1
F20260927 21:15:51.755066 16859 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1ac0e079ad google::LogMessage::Fail() at ??:0
@ 0x7f1ac0e0a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1ac0e074de google::LogMessage::Flush() at ??:0
@ 0x7f1ac0e08a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1ac3574104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1ac3571d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1ac8d611cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1ac8d6148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1ac8d61594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1ac8d27c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1ac8d31ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1ac2cd50d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1ac2cca954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1ac3b2e6db start_thread at ??:0
@ 0x7f1abf90561f clone at ??:0
W20260927 21:15:51.847993 16850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:51.848227 16850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:51.848256 16850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:51.849638 16850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:51.849702 16850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:51.849736 16850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:15:51.851423 16850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36897
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:51.851693 16850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:51.851934 16850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:15:51.858495 16889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:51.858616 16892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:51.859145 16890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:51.859619 16850 server_base.cc:1034] running on GCE node
I20260927 21:15:51.859900 16850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:51.860462 16850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:51.861866 16850 hybrid_clock.cc:648] HybridClock initialized: now 1790543751861851 us; error 60 us; skew 500 ppm
Sep 27 21:15:51 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:51.866086 16850 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:15:51.866792 16850 webserver.cc:460] Webserver started at http://127.0.98.1:36897/ using document root <none> and password file <none>
I20260927 21:15:51.867247 16850 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:51.867331 16850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:51.869549 16850 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:15:51.870596 16899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:51.870821 16850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:51.870908 16850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "5ea660de4d814fad87e39ef67dca8521"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "d774d30e4a018334e3906976c98dc59b"
server_key_iv: "76a59784715e468c2043874b994ae589"
server_key_version: "encryptionkey@0"
I20260927 21:15:51.871415 16850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:51.900990 16850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:51.903906 16850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:51.904068 16850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:51.904351 16850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:15:51.904824 16906 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 21:15:51.906762 16906 ts_tablet_manager.cc:1852] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 21:15:51.907143 16906 ts_tablet_manager.cc:1905] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:15:51.909921 16906 ts_tablet_manager.cc:1918] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:15:51.910024 16906 log.cc:1196] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/wal/wals/d516920a09d5420d8a3f8d269eff0405
I20260927 21:15:51.910295 16906 ts_tablet_manager.cc:1939] T d516920a09d5420d8a3f8d269eff0405 P 5ea660de4d814fad87e39ef67dca8521: Deleting consensus metadata
I20260927 21:15:51.910528 16850 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 21:15:51.910600 16850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 21:15:51.910641 16850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:15:51.910665 16850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:51.916664 16850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44231
I20260927 21:15:51.916766 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44231 every 8 connection(s)
I20260927 21:15:51.917028 16850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:15:51.926939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 16850
I20260927 21:15:51.927242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46619
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46331
--webserver_port=38683
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:15:51.967650 17016 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:15:51.917792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:40043)
0927 21:15:51.917965 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:51.918015 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:15:51.918722 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:15:51.967430 (+ 48708us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
W20260927 21:15:51.967916 17014 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46331 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46331: Client connection negotiation failed: client connection to 127.0.98.62:46331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:15:51.968227 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:51.968123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:49213)
0927 21:15:51.968170 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:51.968207 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 21:15:51.968593 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:51.968514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:54435)
0927 21:15:51.968556 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:51.968577 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:15:51.968701 17014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:15:52.010308 17017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:52.010536 17017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:52.010563 17017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:52.011825 17017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:52.011893 17017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:52.011932 17017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:15:52.013562 17017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38683
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:52.013778 17017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:52.013968 17017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:15:52.018982 17023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:52.018987 17022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:52.018982 17025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:52.019459 17017 server_base.cc:1034] running on GCE node
I20260927 21:15:52.019671 17017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:52.019891 17017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:52.021028 17017 hybrid_clock.cc:648] HybridClock initialized: now 1790543752021013 us; error 31 us; skew 500 ppm
Sep 27 21:15:52 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:52.023659 17017 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:15:52.024204 17017 webserver.cc:460] Webserver started at http://127.0.98.2:38683/ using document root <none> and password file <none>
I20260927 21:15:52.024415 17017 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:52.024459 17017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:52.025899 17017 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 21:15:52.026585 17032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:52.026757 17017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:52.026835 17017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "a58fbe5275644d39aa760b196c290609"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "c9ccdc55a1bdd728426bca2365244613"
server_key_iv: "42254ee8e47220a40e1ed43d8754278f"
server_key_version: "encryptionkey@0"
I20260927 21:15:52.027196 17017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:52.049945 17017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:52.052109 17017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:52.052245 17017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:52.052506 17017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:15:52.052829 17017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:15:52.052862 17017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:52.052891 17017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:15:52.052908 17017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:52.058725 17017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46619
I20260927 21:15:52.058776 17145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46619 every 8 connection(s)
I20260927 21:15:52.059121 17017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:15:52.060187 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:52.059787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:50453)
0927 21:15:52.059934 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:52.060013 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
W20260927 21:15:52.060586 17146 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46331 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46331: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
I20260927 21:15:52.060950 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:52.060829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:40395)
0927 21:15:52.060893 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:52.060927 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 21:15:52.061192 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:52.061138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:48457)
0927 21:15:52.061159 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:52.061178 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 21:15:52.061339 17146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:15:52.062660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 17017
I20260927 21:15:52.062919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36705
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46331
--webserver_port=43949
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:15:52.146147 17149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:15:52.146376 17149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:15:52.146401 17149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:15:52.147621 17149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:15:52.147673 17149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:15:52.147701 17149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:15:52.149236 17149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43949
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:15:52.149497 17149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:15:52.149729 17149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:15:52.154932 17155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:52.154927 17154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:15:52.154989 17157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:15:52.155323 17149 server_base.cc:1034] running on GCE node
I20260927 21:15:52.155516 17149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:15:52.155778 17149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:15:52.156934 17149 hybrid_clock.cc:648] HybridClock initialized: now 1790543752156920 us; error 32 us; skew 500 ppm
Sep 27 21:15:52 dist-test-slave-r9x1 krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:15:52.159885 17149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:15:52.160365 17149 webserver.cc:460] Webserver started at http://127.0.98.3:43949/ using document root <none> and password file <none>
I20260927 21:15:52.160601 17149 fs_manager.cc:359] Metadata directory not provided
I20260927 21:15:52.160646 17149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:15:52.162164 17149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:15:52.162911 17164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:52.163138 17149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:15:52.163254 17149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "66b910ab3ea94b90afd5aaac58169fba"
format_stamp: "Formatted at 2026-09-27 21:15:10 on dist-test-slave-r9x1"
server_key: "a74488d3119bd6ba5737dfd698e92dd3"
server_key_iv: "62f2aab0ac0573ab5653a47d368bc8fb"
server_key_version: "encryptionkey@0"
I20260927 21:15:52.163625 17149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:15:52.183096 17149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:15:52.185406 17149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:15:52.185530 17149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:15:52.185753 17149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:15:52.186021 17149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:15:52.186053 17149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:52.186074 17149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:15:52.186085 17149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:15:52.192787 17149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36705
I20260927 21:15:52.192836 17277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36705 every 8 connection(s)
I20260927 21:15:52.193212 17149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:15:52.194762 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:52.194095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:58071)
0927 21:15:52.194400 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:52.194536 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
W20260927 21:15:52.195389 17278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46331 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46331: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
I20260927 21:15:52.195979 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:52.195765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:40563)
0927 21:15:52.195880 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:52.195935 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:15:52.196564 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:52.196458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:34087)
0927 21:15:52.196510 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:52.196540 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 21:15:52.196719 17278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:15:52.198074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 17149
I20260927 21:15:52.969411 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:52.969120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:44089)
0927 21:15:52.969306 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:52.969349 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260927 21:15:53.061893 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:53.061660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:38329)
0927 21:15:53.061771 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:53.061802 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":75,"threads_started":1}
I20260927 21:15:53.197451 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:53.197140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:33985)
0927 21:15:53.197345 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:53.197389 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260927 21:15:53.970137 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:53.969895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:33301)
0927 21:15:53.970040 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:53.970089 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260927 21:15:54.062599 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:54.062338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:56037)
0927 21:15:54.062504 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:54.062549 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260927 21:15:54.198033 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:54.197840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:51749)
0927 21:15:54.197955 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:54.197987 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260927 21:15:54.970849 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:54.970570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:51435)
0927 21:15:54.970759 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:54.970801 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260927 21:15:55.063467 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:55.063160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:56335)
0927 21:15:55.063365 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:55.063417 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":107,"threads_started":1}
I20260927 21:15:55.198724 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:55.198498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:50915)
0927 21:15:55.198631 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:55.198677 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260927 21:15:55.971565 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:55.971362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:53493)
0927 21:15:55.971483 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:55.971518 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260927 21:15:56.064301 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:56.064058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:60717)
0927 21:15:56.064205 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:56.064251 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260927 21:15:56.199581 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:56.199299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:36643)
0927 21:15:56.199470 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:56.199530 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260927 21:15:56.972332 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:56.972055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:51561)
0927 21:15:56.972237 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:56.972283 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260927 21:15:57.065018 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:57.064804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:47327)
0927 21:15:57.064930 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:57.064970 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260927 21:15:57.200551 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:57.200125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:60319)
0927 21:15:57.200410 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:57.200482 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
I20260927 21:15:57.973202 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:57.972919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:54525)
0927 21:15:57.973107 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:57.973154 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260927 21:15:58.065809 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:58.065552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:47123)
0927 21:15:58.065715 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:58.065760 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260927 21:15:58.201401 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:58.201149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:50121)
0927 21:15:58.201321 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:58.201353 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260927 21:15:58.973999 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:58.973747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:41645)
0927 21:15:58.973896 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:58.973946 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260927 21:15:59.066579 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:59.066303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:43623)
0927 21:15:59.066484 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:59.066529 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260927 21:15:59.202076 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:59.201868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:35769)
0927 21:15:59.201985 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:59.202028 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260927 21:15:59.974723 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:15:59.974489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:42705)
0927 21:15:59.974630 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:15:59.974674 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260927 21:16:00.067431 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:00.067061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:46813)
0927 21:16:00.067236 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:00.067337 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
I20260927 21:16:00.202885 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:00.202636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:41627)
0927 21:16:00.202785 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:00.202836 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260927 21:16:00.975481 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:00.975269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:48397)
0927 21:16:00.975390 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:00.975434 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260927 21:16:01.068163 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:01.067953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:60973)
0927 21:16:01.068075 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:01.068105 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260927 21:16:01.203759 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:01.203516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:59679)
0927 21:16:01.203665 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:01.203710 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260927 21:16:01.976245 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:01.976020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:36189)
0927 21:16:01.976157 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:01.976194 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260927 21:16:02.068849 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:02.068647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:43225)
0927 21:16:02.068762 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:02.068801 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260927 21:16:02.204732 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:02.204453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:58175)
0927 21:16:02.204639 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:02.204684 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260927 21:16:02.976967 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:02.976714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:33885)
0927 21:16:02.976868 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:02.976910 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260927 21:16:03.069636 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:03.069409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:59081)
0927 21:16:03.069534 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:03.069581 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260927 21:16:03.205498 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:03.205239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:41353)
0927 21:16:03.205404 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:03.205442 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260927 21:16:03.977761 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:03.977504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:37029)
0927 21:16:03.977662 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:03.977711 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260927 21:16:04.070372 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:04.070143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:34381)
0927 21:16:04.070270 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:04.070324 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260927 21:16:04.206291 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:04.206040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:33445)
0927 21:16:04.206186 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:04.206241 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260927 21:16:04.978734 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:04.978464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:36359)
0927 21:16:04.978626 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:04.978683 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260927 21:16:05.071062 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:05.070856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:41609)
0927 21:16:05.070971 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:05.071014 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260927 21:16:05.207136 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:05.206838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:51087)
0927 21:16:05.207034 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:05.207087 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260927 21:16:05.979581 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:05.979281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:36515)
0927 21:16:05.979468 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:05.979520 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260927 21:16:06.071802 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:06.071544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:33415)
0927 21:16:06.071708 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:06.071755 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260927 21:16:06.208021 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:06.207752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:44385)
0927 21:16:06.207938 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:06.207971 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260927 21:16:06.980556 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:06.980303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:51673)
0927 21:16:06.980460 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:06.980507 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
I20260927 21:16:07.072636 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:07.072391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:54891)
0927 21:16:07.072539 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:07.072581 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260927 21:16:07.205610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 16850
I20260927 21:16:07.208868 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:07.208652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:48591)
0927 21:16:07.208790 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:07.208822 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260927 21:16:07.209878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 17017
I20260927 21:16:07.213862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 17149
2026-09-27T21:16:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58553 ms)
[----------] 30 tests from SecurityITest (237491 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:07 dist-test-slave-r9x1 krb5kdc[17329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:07 dist-test-slave-r9x1 krb5kdc[17329](info): set up 2 sockets
Sep 27 21:16:07 dist-test-slave-r9x1 krb5kdc[17329](info): commencing operation
krb5kdc: starting...
W20260927 21:16:08.193068 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:08 dist-test-slave-r9x1 krb5kdc[17329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:08.318835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44515
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:08.403532 17342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:08.403770 17342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:08.403805 17342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:08.405112 17342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:08.405172 17342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:08.405192 17342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:08.405210 17342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:08.405224 17342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:08.406937 17342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:08.407223 17342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:08.407452 17342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:08.414166 17348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:08.414158 17347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:08.414177 17342 server_base.cc:1034] running on GCE node
W20260927 21:16:08.414158 17350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:08.414877 17342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:08.415313 17342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:08.416515 17342 hybrid_clock.cc:648] HybridClock initialized: now 1790543768416495 us; error 37 us; skew 500 ppm
Sep 27 21:16:08 dist-test-slave-r9x1 krb5kdc[17329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:08.419291 17342 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:08.419757 17342 webserver.cc:460] Webserver started at http://127.0.98.62:33293/ using document root <none> and password file <none>
I20260927 21:16:08.419981 17342 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:08.420030 17342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:08.420163 17342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:08.421238 17342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "e21b7c8bcaed4f928e52fadb2365b0e2"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "cfca4037ff21d8e7298be76244594a57"
server_key_iv: "e6281c24523154aa48f63d37cc74e54c"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.421635 17342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "e21b7c8bcaed4f928e52fadb2365b0e2"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "cfca4037ff21d8e7298be76244594a57"
server_key_iv: "e6281c24523154aa48f63d37cc74e54c"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.423012 17342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:08.423831 17357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.424063 17342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:08.424165 17342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "e21b7c8bcaed4f928e52fadb2365b0e2"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "cfca4037ff21d8e7298be76244594a57"
server_key_iv: "e6281c24523154aa48f63d37cc74e54c"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.424247 17342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:08.441715 17342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:08.443965 17342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:08.444089 17342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:08.448002 17342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44515
I20260927 21:16:08.448150 17409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44515 every 8 connection(s)
I20260927 21:16:08.448410 17342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:08.449350 17410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:08.451736 17410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2: Bootstrap starting.
I20260927 21:16:08.452363 17410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:08.452656 17410 log.cc:824] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:08.453504 17410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2: No bootstrap required, opened a new log
I20260927 21:16:08.454698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 17342
I20260927 21:16:08.454787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:08.454969 392 external_mini_cluster.cc:1428] Setting key e5e06a1dd50bf2cd03a1cd486e73607d
I20260927 21:16:08.455353 17410 raft_consensus.cc:348] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21b7c8bcaed4f928e52fadb2365b0e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44515 } }
I20260927 21:16:08.455575 17410 raft_consensus.cc:374] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:08.455641 17410 raft_consensus.cc:729] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21b7c8bcaed4f928e52fadb2365b0e2, State: Initialized, Role: FOLLOWER
I20260927 21:16:08.455791 17410 consensus_queue.cc:260] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21b7c8bcaed4f928e52fadb2365b0e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44515 } }
I20260927 21:16:08.455883 17410 raft_consensus.cc:388] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:08.455916 17410 raft_consensus.cc:482] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:08.455960 17410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:08.457146 17410 raft_consensus.cc:504] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21b7c8bcaed4f928e52fadb2365b0e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44515 } }
I20260927 21:16:08.457258 17410 leader_election.cc:302] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e21b7c8bcaed4f928e52fadb2365b0e2; no voters:
I20260927 21:16:08.457471 17410 leader_election.cc:288] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:08.457526 17415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:08.457659 17415 raft_consensus.cc:686] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [term 1 LEADER]: Becoming Leader. State: Replica: e21b7c8bcaed4f928e52fadb2365b0e2, State: Running, Role: LEADER
I20260927 21:16:08.457755 17410 sys_catalog.cc:564] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:08.457805 17415 consensus_queue.cc:237] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21b7c8bcaed4f928e52fadb2365b0e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44515 } }
I20260927 21:16:08.458303 17417 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e21b7c8bcaed4f928e52fadb2365b0e2. Latest consensus state: current_term: 1 leader_uuid: "e21b7c8bcaed4f928e52fadb2365b0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21b7c8bcaed4f928e52fadb2365b0e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44515 } } }
I20260927 21:16:08.458408 17417 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:08.458312 17416 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e21b7c8bcaed4f928e52fadb2365b0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21b7c8bcaed4f928e52fadb2365b0e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44515 } } }
I20260927 21:16:08.458612 17416 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2 [sys.catalog]: This master's current role is: LEADER
W20260927 21:16:08.459698 17431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:16:08.459760 17431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:08.459815 17422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:08.459990 17422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:08.461738 17422 catalog_manager.cc:1347] Generated new cluster ID: 0d3f525277784f63bd1afae0f63da2ec
I20260927 21:16:08.461794 17422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:16:08 dist-test-slave-r9x1 krb5kdc[17329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:08.466657 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.455533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.0.98.62:44515)
0927 21:16:08.455741 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:08.455751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:08.455783 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:16:08.455886 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:08.455892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:08.455940 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:08.456074 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:08.456847 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.458091 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.459168 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.459783 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.460221 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:08.462738 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:08.462742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:08.462742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:08.462755 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:08.464776 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.465318 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.465321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.465321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.465465 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.465681 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.465682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.465682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.465999 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:08.466148 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:08.466278 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:16:08.466428 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260927 21:16:08.474684 17422 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:08.475466 17422 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:08.488242 17422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2: Generated new TSK 0
I20260927 21:16:08.488499 17422 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:16:08.526387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44515
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:08.615330 17436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:08.615578 17436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:08.615613 17436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:08.616899 17436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:08.616950 17436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:08.616987 17436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:16:08.618736 17436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:08.619015 17436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:08.619241 17436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:08.624553 17441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:08.624648 17442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:08.624622 17436 server_base.cc:1034] running on GCE node
W20260927 21:16:08.624550 17444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:08.625012 17436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:08.625296 17436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:08.626482 17436 hybrid_clock.cc:648] HybridClock initialized: now 1790543768626444 us; error 53 us; skew 500 ppm
Sep 27 21:16:08 dist-test-slave-r9x1 krb5kdc[17329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:08.629247 17436 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:16:08.629751 17436 webserver.cc:460] Webserver started at http://127.0.98.1:38749/ using document root <none> and password file <none>
I20260927 21:16:08.629972 17436 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:08.630021 17436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:08.630153 17436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:08.631214 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "be9baf75fc5e40f7b8768b085fc15f13"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "4fe81a67cde05672d8c99378eea3977d"
server_key_iv: "8e2562dfd2045ecccc3f0303f955c4f8"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.631577 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "be9baf75fc5e40f7b8768b085fc15f13"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "4fe81a67cde05672d8c99378eea3977d"
server_key_iv: "8e2562dfd2045ecccc3f0303f955c4f8"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.633015 17436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:16:08.633961 17451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.634182 17436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:16:08.634274 17436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "be9baf75fc5e40f7b8768b085fc15f13"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "4fe81a67cde05672d8c99378eea3977d"
server_key_iv: "8e2562dfd2045ecccc3f0303f955c4f8"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.634341 17436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:08.644893 17436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:08.647255 17436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:08.647406 17436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:08.647647 17436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:08.647956 17436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:08.647990 17436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.648020 17436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:08.648038 17436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.654399 17436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41293
I20260927 21:16:08.654474 17564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41293 every 8 connection(s)
I20260927 21:16:08.654778 17436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:16:08 dist-test-slave-r9x1 krb5kdc[17329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:08.662535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 17436
I20260927 21:16:08.662642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:16:08.662838 392 external_mini_cluster.cc:1428] Setting key 65c2304de7ca7c58f2e3b952c489bd57
I20260927 21:16:08.665688 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.655531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41779 (local address 127.0.98.62:44515)
0927 21:16:08.655608 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:08.655610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:08.655841 (+ 231us) server_negotiation.cc:407] Connection header received
0927 21:16:08.656730 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:08.656732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:08.656747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:08.656802 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:08.657666 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.658193 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.659404 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.659632 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.659799 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:08.662408 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:08.662409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:08.662410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:08.662420 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:08.664047 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.664584 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.664585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.664586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.664644 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.664916 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.664917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.664917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.665055 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:08.665120 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:08.665483 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 21:16:08.665596 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:16:08.665871 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.655493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44515 (local address 127.0.98.1:41779)
0927 21:16:08.655655 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:08.655695 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:08.656617 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:08.656863 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:08.656867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:08.657148 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:08.657444 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:08.657449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.658311 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.658312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:08.659300 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:08.659304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.659881 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.659882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:08.660036 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:08.660780 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:08.660790 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:08.662293 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:08.664151 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:08.664164 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:08.664168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:08.664498 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:08.664709 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:08.664709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:08.664710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:08.664857 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:08.665129 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:08.665139 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:08.665340 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 21:16:08.665449 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:16:08.665639 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260927 21:16:08.666718 17565 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44515
I20260927 21:16:08.666827 17565 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:08.667037 17565 heartbeater.cc:511] Master 127.0.98.62:44515 requested a full tablet report, sending...
I20260927 21:16:08.667711 17374 ts_manager.cc:194] Registered new tserver with Master: be9baf75fc5e40f7b8768b085fc15f13 (127.0.98.1:41293)
I20260927 21:16:08.668373 17374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41779
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:16:08.693836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44515
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:08.775414 17570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:08.775627 17570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:08.775660 17570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:08.776875 17570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:08.776928 17570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:08.776964 17570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:16:08.778600 17570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:08.778847 17570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:08.779072 17570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:08.784169 17576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:08.784152 17578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:08.784224 17570 server_base.cc:1034] running on GCE node
W20260927 21:16:08.784169 17575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:08.784582 17570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:08.784839 17570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:08.786039 17570 hybrid_clock.cc:648] HybridClock initialized: now 1790543768786010 us; error 30 us; skew 500 ppm
Sep 27 21:16:08 dist-test-slave-r9x1 krb5kdc[17329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:08.788692 17570 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:16:08.789189 17570 webserver.cc:460] Webserver started at http://127.0.98.2:33765/ using document root <none> and password file <none>
I20260927 21:16:08.789428 17570 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:08.789480 17570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:08.789608 17570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:08.790723 17570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "99877f0e384240678d834e6947b001e7"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "646c669d8329f5be62093459234551af"
server_key_iv: "902158d59fa837f9cc2bda569b260098"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.791069 17570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "99877f0e384240678d834e6947b001e7"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "646c669d8329f5be62093459234551af"
server_key_iv: "902158d59fa837f9cc2bda569b260098"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.792460 17570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:08.793352 17585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.793548 17570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:08.793641 17570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "99877f0e384240678d834e6947b001e7"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "646c669d8329f5be62093459234551af"
server_key_iv: "902158d59fa837f9cc2bda569b260098"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.793717 17570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:08.803037 17570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:08.805243 17570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:08.805419 17570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:08.805655 17570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:08.805953 17570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:08.805986 17570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.806016 17570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:08.806038 17570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.811888 17570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36533
I20260927 21:16:08.812008 17698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36533 every 8 connection(s)
I20260927 21:16:08.812291 17570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:16:08 dist-test-slave-r9x1 krb5kdc[17329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:08.818714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 17570
I20260927 21:16:08.818809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:16:08.819022 392 external_mini_cluster.cc:1428] Setting key 4e464cb7a903df9448231e73096f7b85
I20260927 21:16:08.822363 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.813043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33089 (local address 127.0.98.62:44515)
0927 21:16:08.813118 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:08.813120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:08.813313 (+ 193us) server_negotiation.cc:407] Connection header received
0927 21:16:08.814299 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:08.814300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:08.814311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:08.814388 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:08.815118 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.815644 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.816687 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.816876 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.816926 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:08.819245 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:08.819247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:08.819248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:08.819255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:08.820588 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.821115 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.821116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.821117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.821189 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.821488 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.821489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.821489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.821638 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:08.821704 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:08.822098 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 21:16:08.822273 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:16:08.822388 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.812991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44515 (local address 127.0.98.2:33089)
0927 21:16:08.813150 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:08.813218 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:08.814187 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:08.814500 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:08.814505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:08.814786 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:08.814996 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:08.815001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.815766 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.815768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:08.816608 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:08.816611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.816973 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.816974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:08.817088 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:08.817615 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:08.817623 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:08.819121 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:08.820687 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:08.820697 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:08.820699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:08.821021 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:08.821302 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:08.821303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:08.821303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:08.821428 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:08.821697 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:08.821700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:08.821934 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 21:16:08.822037 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:16:08.822193 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260927 21:16:08.822962 17699 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44515
I20260927 21:16:08.823079 17699 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:08.823386 17699 heartbeater.cc:511] Master 127.0.98.62:44515 requested a full tablet report, sending...
I20260927 21:16:08.823894 17374 ts_manager.cc:194] Registered new tserver with Master: 99877f0e384240678d834e6947b001e7 (127.0.98.2:36533)
I20260927 21:16:08.824293 17374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33089
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:16:08.848603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44515
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:08.931630 17704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:08.931847 17704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:08.931875 17704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:08.933211 17704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:08.933310 17704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:08.933353 17704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:16:08.935129 17704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:08.935353 17704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:08.935562 17704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:08.940661 17712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:08.940682 17704 server_base.cc:1034] running on GCE node
W20260927 21:16:08.940665 17710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:08.940665 17709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:08.941156 17704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:08.941435 17704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:08.942589 17704 hybrid_clock.cc:648] HybridClock initialized: now 1790543768942572 us; error 34 us; skew 500 ppm
Sep 27 21:16:08 dist-test-slave-r9x1 krb5kdc[17329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:08.945381 17704 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:16:08.945828 17704 webserver.cc:460] Webserver started at http://127.0.98.3:44307/ using document root <none> and password file <none>
I20260927 21:16:08.946033 17704 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:08.946072 17704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:08.946165 17704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:08.947166 17704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "cde75dd34f70cd77d428f6f6d549d983"
server_key_iv: "1eae97de2dee9b7b76a8ab1696c98d45"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.947446 17704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "cde75dd34f70cd77d428f6f6d549d983"
server_key_iv: "1eae97de2dee9b7b76a8ab1696c98d45"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.948683 17704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:08.949452 17719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.949645 17704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:08.949719 17704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da"
format_stamp: "Formatted at 2026-09-27 21:16:08 on dist-test-slave-r9x1"
server_key: "cde75dd34f70cd77d428f6f6d549d983"
server_key_iv: "1eae97de2dee9b7b76a8ab1696c98d45"
server_key_version: "encryptionkey@0"
I20260927 21:16:08.949806 17704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:08.962832 17704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:08.964864 17704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:08.964987 17704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:08.965235 17704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:08.965583 17704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:08.965618 17704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.965648 17704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:08.965664 17704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:08.971266 17704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36411
I20260927 21:16:08.971329 17832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36411 every 8 connection(s)
I20260927 21:16:08.971626 17704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:16:08.973594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 17704
I20260927 21:16:08.973691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:16:08.973857 392 external_mini_cluster.cc:1428] Setting key e7cd77f9655ae75dfe02dcdcff63f3a9
Sep 27 21:16:08 dist-test-slave-r9x1 krb5kdc[17329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:08.980801 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.972353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47101 (local address 127.0.98.62:44515)
0927 21:16:08.972424 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:08.972427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:08.972600 (+ 173us) server_negotiation.cc:407] Connection header received
0927 21:16:08.973598 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:08.973600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:08.973613 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:08.973660 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:08.974356 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.974850 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.975847 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.976046 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.976112 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:08.978226 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:08.978227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:08.978228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:08.978236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:08.979413 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.979840 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.979842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.979842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.979913 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.980154 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.980155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.980156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.980272 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:08.980329 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:08.980650 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 21:16:08.980729 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:16:08.981005 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.972302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44515 (local address 127.0.98.3:47101)
0927 21:16:08.972478 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:08.972515 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:08.973493 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:08.973766 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:08.973770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:08.974056 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:08.974260 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:08.974265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.974945 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.974946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:08.975758 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:08.975761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.976126 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.976127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:08.976246 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:08.976717 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:08.976726 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:08.978126 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:08.979503 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:08.979511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:08.979513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:08.979743 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:08.979978 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:08.979979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:08.979979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:08.980091 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:08.980336 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:08.980342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:08.980491 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 21:16:08.980617 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 21:16:08.980781 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260927 21:16:08.981514 17833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44515
I20260927 21:16:08.981616 17833 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:08.981801 17833 heartbeater.cc:511] Master 127.0.98.62:44515 requested a full tablet report, sending...
I20260927 21:16:08.982206 17374 ts_manager.cc:194] Registered new tserver with Master: 4f62a9c88cb54c28ac39ef8d2bfa45da (127.0.98.3:36411)
I20260927 21:16:08.982550 17374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47101
I20260927 21:16:08.985466 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:16:08.992579 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.986752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.0.98.62:44515)
0927 21:16:08.986816 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:08.986818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:08.986895 (+ 77us) server_negotiation.cc:407] Connection header received
0927 21:16:08.986948 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:08.986949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:08.986957 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:08.987014 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:08.987508 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.987990 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.988885 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.989083 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.989134 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:08.990320 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:08.990321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:08.990322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:08.990329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:08.991534 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.991854 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.991855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.991855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.991904 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:08.992100 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:08.992101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:08.992101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:08.992247 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:08.992319 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:08.992433 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 21:16:08.992511 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:16:08.994180 17374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:16:08.995736 17374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:16:09.002451 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.998355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36411 (local address 127.0.0.1:39048)
0927 21:16:08.998587 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:08.998598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:08.998659 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:08.999007 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:08.999008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:08.999010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:08.999044 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:08.999045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.000305 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.000306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.001432 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.001434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.002111 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.002112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.002200 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.002216 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.002283 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.002342 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":65,"threads_started":1}
I20260927 21:16:09.002503 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.998284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.0.98.2:36533)
0927 21:16:08.998775 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:08.998778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:08.998787 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:08.998823 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:08.998824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:08.998836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:08.999078 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:08.999167 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.999857 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.001374 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.002136 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.002241 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.002298 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.002366 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.002412 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":99,"threads_started":1}
I20260927 21:16:09.002815 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.998581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.0.98.3:36411)
0927 21:16:08.998756 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:08.998760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:08.998769 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:08.998804 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:08.998806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:08.998820 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:08.998912 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:08.999113 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.000176 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.001516 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.002019 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.002086 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.002153 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.002687 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.002735 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260927 21:16:09.002451 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.998172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36533 (local address 127.0.0.1:40898)
0927 21:16:08.998327 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:08.998368 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:08.998431 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:08.998936 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:08.998937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:08.998939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:08.999004 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:08.999006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:08.999971 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:08.999972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.001294 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.001298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.002245 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.002246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.002304 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.002307 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.002353 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.002386 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":52,"threads_started":1}
I20260927 21:16:09.003732 17633 tablet_service.cc:1467] Processing CreateTablet for tablet 364717ce97ff49b2a253239d062dbf70 (DEFAULT_TABLE table=test-table [id=c86b31e58e5840488269c1c078ff8ce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:09.003733 17767 tablet_service.cc:1467] Processing CreateTablet for tablet 364717ce97ff49b2a253239d062dbf70 (DEFAULT_TABLE table=test-table [id=c86b31e58e5840488269c1c078ff8ce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:09.004117 17767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364717ce97ff49b2a253239d062dbf70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:09.004117 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364717ce97ff49b2a253239d062dbf70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:09.007658 17850 tablet_bootstrap.cc:492] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: Bootstrap starting.
I20260927 21:16:09.007920 17851 tablet_bootstrap.cc:492] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: Bootstrap starting.
I20260927 21:16:09.007990 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.998299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.0.98.1:41293)
0927 21:16:08.998525 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:08.998531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:08.998594 (+ 63us) server_negotiation.cc:407] Connection header received
0927 21:16:08.998646 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:08.998649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:08.998664 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:08.998735 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:08.999007 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.000177 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.002504 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.003180 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.003249 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.003310 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.007850 (+ 4540us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.007922 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260927 21:16:09.008256 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:08.998172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41293 (local address 127.0.0.1:55612)
0927 21:16:08.998532 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:08.998541 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:08.998588 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:08.998843 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:08.998847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:08.998849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:08.998928 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:08.998930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.001320 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.001321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.002408 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.002411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.003263 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.003264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.007706 (+ 4442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.007715 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.008131 (+ 416us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.008195 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260927 21:16:09.008569 17851 tablet_bootstrap.cc:654] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:09.009186 17851 log.cc:824] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:09.009143 17499 tablet_service.cc:1467] Processing CreateTablet for tablet 364717ce97ff49b2a253239d062dbf70 (DEFAULT_TABLE table=test-table [id=c86b31e58e5840488269c1c078ff8ce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:09.009533 17499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364717ce97ff49b2a253239d062dbf70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:09.010015 17850 tablet_bootstrap.cc:654] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:09.010205 17851 tablet_bootstrap.cc:492] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: No bootstrap required, opened a new log
I20260927 21:16:09.010280 17851 ts_tablet_manager.cc:1397] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:16:09.010550 17850 log.cc:824] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:09.011211 17850 tablet_bootstrap.cc:492] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: No bootstrap required, opened a new log
I20260927 21:16:09.011271 17850 ts_tablet_manager.cc:1397] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:16:09.012271 17851 raft_consensus.cc:348] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } }
I20260927 21:16:09.012473 17851 raft_consensus.cc:374] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:09.012511 17851 raft_consensus.cc:729] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f62a9c88cb54c28ac39ef8d2bfa45da, State: Initialized, Role: FOLLOWER
I20260927 21:16:09.012686 17851 consensus_queue.cc:260] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } }
I20260927 21:16:09.012773 17850 raft_consensus.cc:348] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } }
I20260927 21:16:09.012993 17850 raft_consensus.cc:374] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:09.013058 17850 raft_consensus.cc:729] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99877f0e384240678d834e6947b001e7, State: Initialized, Role: FOLLOWER
I20260927 21:16:09.013055 17854 tablet_bootstrap.cc:492] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13: Bootstrap starting.
I20260927 21:16:09.013085 17833 heartbeater.cc:503] Master 127.0.98.62:44515 was elected leader, sending a full tablet report...
I20260927 21:16:09.013226 17851 ts_tablet_manager.cc:1428] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:16:09.013212 17850 consensus_queue.cc:260] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } }
I20260927 21:16:09.013568 17850 ts_tablet_manager.cc:1428] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:16:09.013682 17699 heartbeater.cc:503] Master 127.0.98.62:44515 was elected leader, sending a full tablet report...
I20260927 21:16:09.013769 17854 tablet_bootstrap.cc:654] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:09.014379 17854 log.cc:824] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:09.015039 17854 tablet_bootstrap.cc:492] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13: No bootstrap required, opened a new log
I20260927 21:16:09.015095 17854 ts_tablet_manager.cc:1397] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:16:09.016443 17854 raft_consensus.cc:348] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } }
I20260927 21:16:09.016584 17854 raft_consensus.cc:374] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:09.016605 17854 raft_consensus.cc:729] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9baf75fc5e40f7b8768b085fc15f13, State: Initialized, Role: FOLLOWER
I20260927 21:16:09.016702 17854 consensus_queue.cc:260] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } }
I20260927 21:16:09.016961 17854 ts_tablet_manager.cc:1428] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:16:09.016978 17565 heartbeater.cc:503] Master 127.0.98.62:44515 was elected leader, sending a full tablet report...
W20260927 21:16:09.062836 17700 tablet.cc:2369] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:16:09.112244 17855 raft_consensus.cc:482] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:16:09.112378 17855 raft_consensus.cc:504] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } }
I20260927 21:16:09.112720 17855 leader_election.cc:288] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99877f0e384240678d834e6947b001e7 (127.0.98.2:36533), be9baf75fc5e40f7b8768b085fc15f13 (127.0.98.1:41293)
I20260927 21:16:09.116237 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.113005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51333 (local address 127.0.98.2:36533)
0927 21:16:09.113050 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.113051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.113065 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:16:09.113100 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.113101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.113113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.113176 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:09.113450 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.114226 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.115506 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.115897 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.115961 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.115974 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.116124 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.116168 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:16:09.116247 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.113049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39835 (local address 127.0.98.1:41293)
0927 21:16:09.113092 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.113094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.113216 (+ 122us) server_negotiation.cc:407] Connection header received
0927 21:16:09.113255 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.113256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.113289 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.113344 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:09.113514 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.114248 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.115503 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.115893 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.115953 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.115964 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.116129 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.116181 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:16:09.116241 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.112923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36533 (local address 127.0.98.3:51333)
0927 21:16:09.112984 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.112997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.113040 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.113238 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.113239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.113241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:09.113361 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.113364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.114360 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.114362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.115406 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.115409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.115981 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.115981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.116051 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.116054 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.116099 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.116145 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":71}
I20260927 21:16:09.116264 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.112924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41293 (local address 127.0.98.3:39835)
0927 21:16:09.113133 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.113141 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.113202 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.113384 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.113386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.113388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:09.113447 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.113449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.114362 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.114364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.115403 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.115407 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.115982 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.115983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.116060 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.116064 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.116111 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.116155 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":82,"thread_start_us":106,"threads_started":1}
I20260927 21:16:09.116653 17653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364717ce97ff49b2a253239d062dbf70" candidate_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99877f0e384240678d834e6947b001e7" is_pre_election: true
I20260927 21:16:09.116698 17519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364717ce97ff49b2a253239d062dbf70" candidate_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9baf75fc5e40f7b8768b085fc15f13" is_pre_election: true
I20260927 21:16:09.116819 17653 raft_consensus.cc:2454] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f62a9c88cb54c28ac39ef8d2bfa45da in term 0.
I20260927 21:16:09.116837 17519 raft_consensus.cc:2454] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f62a9c88cb54c28ac39ef8d2bfa45da in term 0.
I20260927 21:16:09.117082 17722 leader_election.cc:302] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f62a9c88cb54c28ac39ef8d2bfa45da, be9baf75fc5e40f7b8768b085fc15f13; no voters:
I20260927 21:16:09.117233 17855 raft_consensus.cc:2781] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:16:09.117321 17855 raft_consensus.cc:482] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:16:09.117347 17855 raft_consensus.cc:3037] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:09.118024 17855 raft_consensus.cc:504] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } }
I20260927 21:16:09.118157 17855 leader_election.cc:288] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [CANDIDATE]: Term 1 election: Requested vote from peers 99877f0e384240678d834e6947b001e7 (127.0.98.2:36533), be9baf75fc5e40f7b8768b085fc15f13 (127.0.98.1:41293)
I20260927 21:16:09.118361 17653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364717ce97ff49b2a253239d062dbf70" candidate_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99877f0e384240678d834e6947b001e7"
I20260927 21:16:09.118373 17519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364717ce97ff49b2a253239d062dbf70" candidate_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9baf75fc5e40f7b8768b085fc15f13"
I20260927 21:16:09.118440 17653 raft_consensus.cc:3037] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:09.118458 17519 raft_consensus.cc:3037] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:09.119148 17653 raft_consensus.cc:2454] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f62a9c88cb54c28ac39ef8d2bfa45da in term 1.
I20260927 21:16:09.119256 17519 raft_consensus.cc:2454] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f62a9c88cb54c28ac39ef8d2bfa45da in term 1.
I20260927 21:16:09.119390 17721 leader_election.cc:302] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f62a9c88cb54c28ac39ef8d2bfa45da, 99877f0e384240678d834e6947b001e7; no voters:
I20260927 21:16:09.119508 17855 raft_consensus.cc:2781] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:09.119673 17855 raft_consensus.cc:686] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 1 LEADER]: Becoming Leader. State: Replica: 4f62a9c88cb54c28ac39ef8d2bfa45da, State: Running, Role: LEADER
I20260927 21:16:09.119760 17855 consensus_queue.cc:237] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } }
I20260927 21:16:09.120625 17373 catalog_manager.cc:5495] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f62a9c88cb54c28ac39ef8d2bfa45da (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f62a9c88cb54c28ac39ef8d2bfa45da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36411 } health_report { overall_health: HEALTHY } } }
I20260927 21:16:09.148663 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.146305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.0.98.3:36411)
0927 21:16:09.146390 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.146392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.146399 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:16:09.146433 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.146434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.146451 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.146509 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:09.146656 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.147136 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.147902 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.148090 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.148147 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.148307 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:09.148423 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:09.148497 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.148540 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.148562 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:16:09.150668 17519 raft_consensus.cc:1261] T 364717ce97ff49b2a253239d062dbf70 P be9baf75fc5e40f7b8768b085fc15f13 [term 1 FOLLOWER]: Refusing update from remote peer 4f62a9c88cb54c28ac39ef8d2bfa45da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:09.150668 17653 raft_consensus.cc:1261] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [term 1 FOLLOWER]: Refusing update from remote peer 4f62a9c88cb54c28ac39ef8d2bfa45da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:09.151098 17855 consensus_queue.cc:1035] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [LEADER]: Connected to new peer: Peer: permanent_uuid: "99877f0e384240678d834e6947b001e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:09.151265 17860 consensus_queue.cc:1035] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9baf75fc5e40f7b8768b085fc15f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:09.153898 17864 mvcc.cc:205] Tried to move back new op lower bound from 7334067278437986304 to 7334067278315298816. Current Snapshot: MvccSnapshot[applied={T|T < 7334067278437986304}]
I20260927 21:16:09.154503 17865 mvcc.cc:205] Tried to move back new op lower bound from 7334067278437986304 to 7334067278315298816. Current Snapshot: MvccSnapshot[applied={T|T < 7334067278437986304 or (T in {7334067278437986304})}]
I20260927 21:16:09.156118 17373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:09.156215 17373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:09.156994 17373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e21b7c8bcaed4f928e52fadb2365b0e2: Sending DeleteTablet for 3 replicas of tablet 364717ce97ff49b2a253239d062dbf70
I20260927 21:16:09.157406 17767 tablet_service.cc:1514] Processing DeleteTablet for tablet 364717ce97ff49b2a253239d062dbf70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39048
I20260927 21:16:09.157474 17633 tablet_service.cc:1514] Processing DeleteTablet for tablet 364717ce97ff49b2a253239d062dbf70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40898
I20260927 21:16:09.157732 17873 tablet_replica.cc:330] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: stopping tablet replica
I20260927 21:16:09.157734 17499 tablet_service.cc:1514] Processing DeleteTablet for tablet 364717ce97ff49b2a253239d062dbf70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55612
I20260927 21:16:09.157843 17873 raft_consensus.cc:2229] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:16:09.157950 17873 raft_consensus.cc:2258] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:09.157977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 17436
I20260927 21:16:09.158906 17872 tablet_replica.cc:330] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: stopping tablet replica
I20260927 21:16:09.159006 17872 raft_consensus.cc:2229] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:16:09.159101 17872 raft_consensus.cc:2258] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:09.159174 17873 ts_tablet_manager.cc:1905] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:09.159518 17872 ts_tablet_manager.cc:1905] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:09.161408 17872 ts_tablet_manager.cc:1918] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:09.161408 17873 ts_tablet_manager.cc:1918] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:09.161494 17873 log.cc:1196] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/364717ce97ff49b2a253239d062dbf70
I20260927 21:16:09.161618 17872 log.cc:1196] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/364717ce97ff49b2a253239d062dbf70
I20260927 21:16:09.161813 17873 ts_tablet_manager.cc:1939] T 364717ce97ff49b2a253239d062dbf70 P 99877f0e384240678d834e6947b001e7: Deleting consensus metadata
I20260927 21:16:09.161962 17872 ts_tablet_manager.cc:1939] T 364717ce97ff49b2a253239d062dbf70 P 4f62a9c88cb54c28ac39ef8d2bfa45da: Deleting consensus metadata
I20260927 21:16:09.162384 17359 catalog_manager.cc:4841] TS 99877f0e384240678d834e6947b001e7 (127.0.98.2:36533): tablet 364717ce97ff49b2a253239d062dbf70 (table test-table [id=c86b31e58e5840488269c1c078ff8ce5]) successfully deleted
I20260927 21:16:09.162526 17360 catalog_manager.cc:4841] TS 4f62a9c88cb54c28ac39ef8d2bfa45da (127.0.98.3:36411): tablet 364717ce97ff49b2a253239d062dbf70 (table test-table [id=c86b31e58e5840488269c1c078ff8ce5]) successfully deleted
W20260927 21:16:09.164316 17360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41293 (error 108)
I20260927 21:16:09.164510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 17570
I20260927 21:16:09.164916 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.164805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41293 (local address 127.0.0.1:55626)
0927 21:16:09.164856 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.164882 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 21:16:09.165035 17360 catalog_manager.cc:4584] TS be9baf75fc5e40f7b8768b085fc15f13 (127.0.98.1:41293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 364717ce97ff49b2a253239d062dbf70: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41293: connect: Connection refused (error 111)
I20260927 21:16:09.169159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 17704
I20260927 21:16:09.173672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 17342
2026-09-27T21:16:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1959 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T21:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:09Z Disabled control of system clock
I20260927 21:16:09.200366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46343
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:16:09.280674 17879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:09.280900 17879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:09.280920 17879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:09.282142 17879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:09.282188 17879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:09.282199 17879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:09.282210 17879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:09.282222 17879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:09.283561 17879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:09.283774 17879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:09.284030 17879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:09.288853 17885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:09.288908 17887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:09.288940 17879 server_base.cc:1034] running on GCE node
W20260927 21:16:09.288854 17884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:09.289525 17879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:09.289827 17879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:09.290994 17879 hybrid_clock.cc:648] HybridClock initialized: now 1790543769290974 us; error 36 us; skew 500 ppm
I20260927 21:16:09.292238 17879 webserver.cc:460] Webserver started at http://127.0.98.62:45217/ using document root <none> and password file <none>
I20260927 21:16:09.292482 17879 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:09.292533 17879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:09.292665 17879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:09.293790 17879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "e4b918865be34d0a9b5b667cd399f5c2"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "f9aa35ae9a0cb25b0f300be87f3b95cc"
server_key_iv: "43263b96c91c9f3edde958b118f502a9"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.294157 17879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4b918865be34d0a9b5b667cd399f5c2"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "f9aa35ae9a0cb25b0f300be87f3b95cc"
server_key_iv: "43263b96c91c9f3edde958b118f502a9"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.295468 17879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:16:09.296315 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.296520 17879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:09.296610 17879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "e4b918865be34d0a9b5b667cd399f5c2"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "f9aa35ae9a0cb25b0f300be87f3b95cc"
server_key_iv: "43263b96c91c9f3edde958b118f502a9"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.296680 17879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:09.305696 17879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:09.306022 17879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:09.306159 17879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:09.310005 17879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46343
I20260927 21:16:09.310045 17945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46343 every 8 connection(s)
I20260927 21:16:09.310398 17879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:09.311403 17946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:09.313796 17946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2: Bootstrap starting.
I20260927 21:16:09.314379 17946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:09.314688 17946 log.cc:824] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:09.314795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 17879
I20260927 21:16:09.314888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:09.315079 392 external_mini_cluster.cc:1428] Setting key d3801f84b0269871251a21c25511bfe6
I20260927 21:16:09.315495 17946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2: No bootstrap required, opened a new log
I20260927 21:16:09.318105 17946 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b918865be34d0a9b5b667cd399f5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46343 } }
I20260927 21:16:09.318291 17946 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:09.318338 17946 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b918865be34d0a9b5b667cd399f5c2, State: Initialized, Role: FOLLOWER
I20260927 21:16:09.318466 17946 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b918865be34d0a9b5b667cd399f5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46343 } }
I20260927 21:16:09.318542 17946 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:09.318579 17946 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:09.318622 17946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:09.319342 17946 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b918865be34d0a9b5b667cd399f5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46343 } }
I20260927 21:16:09.319449 17946 leader_election.cc:302] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b918865be34d0a9b5b667cd399f5c2; no voters:
I20260927 21:16:09.319691 17946 leader_election.cc:288] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:09.319731 17951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:09.319990 17951 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [term 1 LEADER]: Becoming Leader. State: Replica: e4b918865be34d0a9b5b667cd399f5c2, State: Running, Role: LEADER
I20260927 21:16:09.320137 17951 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b918865be34d0a9b5b667cd399f5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46343 } }
I20260927 21:16:09.320413 17946 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:09.320634 17953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4b918865be34d0a9b5b667cd399f5c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b918865be34d0a9b5b667cd399f5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46343 } } }
I20260927 21:16:09.320876 17953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:09.320814 17952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b918865be34d0a9b5b667cd399f5c2. Latest consensus state: current_term: 1 leader_uuid: "e4b918865be34d0a9b5b667cd399f5c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b918865be34d0a9b5b667cd399f5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46343 } } }
I20260927 21:16:09.321172 17952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:09.321389 17958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:09.321568 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.315676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.0.98.62:46343)
0927 21:16:09.315883 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.315891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.315916 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:16:09.316719 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.316727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.316789 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.316907 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:09.317167 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.318108 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.319135 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.319668 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.319826 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.320575 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:09.320583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:09.320583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:09.320789 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:09.320835 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:09.320853 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:09.320913 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.321218 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.321348 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260927 21:16:09.321873 17958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:09.323560 17958 catalog_manager.cc:1347] Generated new cluster ID: 8536e4f874a34b84815ed3f6b7b16866
I20260927 21:16:09.323622 17958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:09.344511 17958 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:09.345086 17958 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:09.350430 17958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2: Generated new TSK 0
I20260927 21:16:09.350682 17958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:09.353119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46343
--builtin_ntp_servers=127.0.98.20:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:16:09.439143 17970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:09.439404 17970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:09.439440 17970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:09.441457 17970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:09.441519 17970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:09.441561 17970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:16:09.443234 17970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:09.443523 17970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:09.443840 17970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:09.450268 17976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:09.450271 17975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:09.450476 17970 server_base.cc:1034] running on GCE node
W20260927 21:16:09.450325 17978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:09.450860 17970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:09.451125 17970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:09.452353 17970 hybrid_clock.cc:648] HybridClock initialized: now 1790543769452322 us; error 43 us; skew 500 ppm
I20260927 21:16:09.453806 17970 webserver.cc:460] Webserver started at http://127.0.98.1:40263/ using document root <none> and password file <none>
I20260927 21:16:09.454047 17970 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:09.454099 17970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:09.454237 17970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:09.455341 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "231e22db7c35408c892ffd24244c0777"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "31c2ef2da2d02beed2ac6ba4348c5090"
server_key_iv: "0c21d32e95e3e5e59fb5d87ca49eb4f2"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.455713 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "231e22db7c35408c892ffd24244c0777"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "31c2ef2da2d02beed2ac6ba4348c5090"
server_key_iv: "0c21d32e95e3e5e59fb5d87ca49eb4f2"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.457520 17970 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 21:16:09.458820 17984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.459107 17970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:09.459224 17970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "231e22db7c35408c892ffd24244c0777"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "31c2ef2da2d02beed2ac6ba4348c5090"
server_key_iv: "0c21d32e95e3e5e59fb5d87ca49eb4f2"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.459321 17970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:09.470867 17970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:09.471208 17970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:09.471343 17970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:09.471577 17970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:09.471903 17970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:09.471946 17970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.471977 17970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:09.471995 17970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.477845 17970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46381
I20260927 21:16:09.477947 18097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46381 every 8 connection(s)
I20260927 21:16:09.478322 17970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:16:09.478540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 17970
I20260927 21:16:09.478632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:16:09.478845 392 external_mini_cluster.cc:1428] Setting key 1be8c50788fa01c4f886418e1ea67aba
I20260927 21:16:09.479946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46343
--builtin_ntp_servers=127.0.98.20:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:16:09.484992 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.479121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54631 (local address 127.0.98.62:46343)
0927 21:16:09.479172 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.479175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.479504 (+ 329us) server_negotiation.cc:407] Connection header received
0927 21:16:09.480137 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.480138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.480156 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.480220 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:09.480899 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.481614 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.483463 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.483661 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.483726 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.484721 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:09.484722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:09.484723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:09.484728 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:09.484744 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:09.484747 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:09.484794 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.484899 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.484925 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:16:09.485311 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.479137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46343 (local address 127.0.98.1:54631)
0927 21:16:09.479369 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.479419 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.479990 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.480275 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.480279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.480288 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:09.480784 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.480788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.481717 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.481718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.483317 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.483321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.483761 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.483762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.483939 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.484525 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:16:09.484525 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:09.484571 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:16:09.484609 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:09.484815 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:09.484820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:09.484824 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.484901 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.485029 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260927 21:16:09.485875 18098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46343
I20260927 21:16:09.485981 18098 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:09.486212 18098 heartbeater.cc:511] Master 127.0.98.62:46343 requested a full tablet report, sending...
I20260927 21:16:09.486816 17910 ts_manager.cc:194] Registered new tserver with Master: 231e22db7c35408c892ffd24244c0777 (127.0.98.1:46381)
I20260927 21:16:09.487766 17910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54631
W20260927 21:16:09.563930 18101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:09.564324 18101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:09.564390 18101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:09.565891 18101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:09.565982 18101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:09.566015 18101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:16:09.567745 18101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:09.567991 18101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:09.568275 18101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:09.573407 18106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:09.573450 18109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:09.573469 18107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:09.573695 18101 server_base.cc:1034] running on GCE node
I20260927 21:16:09.573930 18101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:09.574184 18101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:09.575364 18101 hybrid_clock.cc:648] HybridClock initialized: now 1790543769575350 us; error 30 us; skew 500 ppm
I20260927 21:16:09.576881 18101 webserver.cc:460] Webserver started at http://127.0.98.2:36417/ using document root <none> and password file <none>
I20260927 21:16:09.577148 18101 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:09.577203 18101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:09.577358 18101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:09.578464 18101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "a1594c5509a040db83e83824a28def2a"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "788e1f1b157e18f2a78b8ff80ad3a843"
server_key_iv: "2710eab4333e316446a75106a2d2e127"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.578804 18101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a1594c5509a040db83e83824a28def2a"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "788e1f1b157e18f2a78b8ff80ad3a843"
server_key_iv: "2710eab4333e316446a75106a2d2e127"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.580132 18101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:09.580976 18115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.581337 18101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:09.581460 18101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "a1594c5509a040db83e83824a28def2a"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "788e1f1b157e18f2a78b8ff80ad3a843"
server_key_iv: "2710eab4333e316446a75106a2d2e127"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.581542 18101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:09.596622 18101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:09.596962 18101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:09.597095 18101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:09.597391 18101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:09.597733 18101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:09.597766 18101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.597796 18101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:09.597815 18101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.603847 18101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39069
I20260927 21:16:09.603888 18228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39069 every 8 connection(s)
I20260927 21:16:09.604228 18101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:16:09.605598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 18101
I20260927 21:16:09.605705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:16:09.605896 392 external_mini_cluster.cc:1428] Setting key 52a435313f5432d88da1a5d220f98269
I20260927 21:16:09.606824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46343
--builtin_ntp_servers=127.0.98.20:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:16:09.610200 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.605033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33643 (local address 127.0.98.62:46343)
0927 21:16:09.605106 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.605108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.605332 (+ 224us) server_negotiation.cc:407] Connection header received
0927 21:16:09.605942 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.605944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.605957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.606001 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:09.606778 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.607310 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.608631 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.608830 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.608897 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.609866 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:09.609868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:09.609868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:09.609873 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:09.609890 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:09.609894 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:09.609946 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.610099 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.610131 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:16:09.610424 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.604996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46343 (local address 127.0.98.2:33643)
0927 21:16:09.605197 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.605232 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.605839 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.606089 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.606093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.606103 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:09.606682 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.606688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.607402 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.607403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.608515 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.608519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.608931 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.608932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.609099 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.609696 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:16:09.609697 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:09.609740 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:16:09.609772 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:09.609961 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:09.609974 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:09.609977 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.610057 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.610184 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260927 21:16:09.610948 18229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46343
I20260927 21:16:09.611068 18229 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:09.611271 18229 heartbeater.cc:511] Master 127.0.98.62:46343 requested a full tablet report, sending...
I20260927 21:16:09.611704 17910 ts_manager.cc:194] Registered new tserver with Master: a1594c5509a040db83e83824a28def2a (127.0.98.2:39069)
I20260927 21:16:09.612106 17910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33643
W20260927 21:16:09.689033 18232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:09.689247 18232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:09.689296 18232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:09.690502 18232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:09.690553 18232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:09.690579 18232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:16:09.691946 18232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:09.692178 18232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:09.692453 18232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:09.697607 18238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:09.697642 18232 server_base.cc:1034] running on GCE node
W20260927 21:16:09.697665 18240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:09.697613 18237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:09.697966 18232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:09.698225 18232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:09.699395 18232 hybrid_clock.cc:648] HybridClock initialized: now 1790543769699377 us; error 33 us; skew 500 ppm
I20260927 21:16:09.700747 18232 webserver.cc:460] Webserver started at http://127.0.98.3:37977/ using document root <none> and password file <none>
I20260927 21:16:09.701026 18232 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:09.701081 18232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:09.701352 18232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:09.702452 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "ef64f71469bc4696b13f6e8b64edb33c"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "b4fdcd3d8835e38871ed7df602f08244"
server_key_iv: "194b8fa799e01732077ea3eab5c2fd04"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.702780 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ef64f71469bc4696b13f6e8b64edb33c"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "b4fdcd3d8835e38871ed7df602f08244"
server_key_iv: "194b8fa799e01732077ea3eab5c2fd04"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.704123 18232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:09.704998 18246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.705193 18232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:09.705299 18232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "ef64f71469bc4696b13f6e8b64edb33c"
format_stamp: "Formatted at 2026-09-27 21:16:09 on dist-test-slave-r9x1"
server_key: "b4fdcd3d8835e38871ed7df602f08244"
server_key_iv: "194b8fa799e01732077ea3eab5c2fd04"
server_key_version: "encryptionkey@0"
I20260927 21:16:09.705368 18232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:09.721132 18232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:09.721503 18232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:09.721654 18232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:09.721915 18232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:09.722257 18232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:09.722292 18232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.722316 18232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:09.722327 18232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:09.727577 18232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36259
I20260927 21:16:09.727650 18359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36259 every 8 connection(s)
I20260927 21:16:09.727979 18232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:16:09.732087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 18232
I20260927 21:16:09.732179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:16:09.732368 392 external_mini_cluster.cc:1428] Setting key 9ed7e717a21fc9a25bc757dc28daa86e
I20260927 21:16:09.733529 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.728751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35277 (local address 127.0.98.62:46343)
0927 21:16:09.728831 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.728833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.729074 (+ 241us) server_negotiation.cc:407] Connection header received
0927 21:16:09.729595 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.729596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.729607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.729670 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:09.730360 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.730926 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.731918 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.732128 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.732208 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.733201 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:09.733203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:09.733203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:09.733207 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:09.733221 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:09.733225 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:09.733323 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.733404 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.733431 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:16:09.733742 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.728699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46343 (local address 127.0.98.3:35277)
0927 21:16:09.728921 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.728969 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.729494 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.729725 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.729728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.729737 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:09.730252 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.730256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.731012 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.731013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.731809 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.731811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.732249 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.732251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.732421 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.733041 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:16:09.733042 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:09.733083 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:16:09.733115 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:09.733329 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:09.733335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:09.733338 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.733406 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.733522 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260927 21:16:09.734193 18360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46343
I20260927 21:16:09.734287 18360 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:09.734472 18360 heartbeater.cc:511] Master 127.0.98.62:46343 requested a full tablet report, sending...
I20260927 21:16:09.734802 17910 ts_manager.cc:194] Registered new tserver with Master: ef64f71469bc4696b13f6e8b64edb33c (127.0.98.3:36259)
I20260927 21:16:09.735183 17910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35277
I20260927 21:16:09.744735 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:16:09.749258 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.746206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.0.98.62:46343)
0927 21:16:09.746306 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.746308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.746391 (+ 83us) server_negotiation.cc:407] Connection header received
0927 21:16:09.746460 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.746461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.746471 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.746519 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:09.746679 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.747198 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.747990 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.748189 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.748248 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.748985 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:09.748987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:09.748987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:09.749013 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:09.749027 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:09.749031 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:09.749074 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.749143 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.749170 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:16:09.751057 17910 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:16:09.752641 17910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:16:09.759315 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.755130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46381 (local address 127.0.0.1:38666)
0927 21:16:09.755421 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.755466 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.755536 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.755909 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.755912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.755914 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:09.755997 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.755998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.756889 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.756890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.758086 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.758089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.758978 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.758980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.759068 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.759084 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.759163 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.759219 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":4,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260927 21:16:09.759403 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.755385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.0.98.1:46381)
0927 21:16:09.755693 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.755696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.755706 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:16:09.755744 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.755746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.755760 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.755834 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:09.756076 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.756792 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.758179 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.758870 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.758953 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.759067 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.759197 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.759314 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":81,"threads_started":1}
I20260927 21:16:09.759693 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.755399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.0.98.2:39069)
0927 21:16:09.755615 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.755620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.755718 (+ 98us) server_negotiation.cc:407] Connection header received
0927 21:16:09.755758 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.755760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.755777 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.755838 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:09.756078 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.757204 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.758414 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.759112 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.759174 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.759364 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.759577 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.759618 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260927 21:16:09.759676 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.755130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39069 (local address 127.0.0.1:47846)
0927 21:16:09.755621 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.755629 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.755669 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.755912 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.755913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.755915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:09.755997 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.756000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.757324 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.757325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.758330 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.758333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.759428 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.759430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.759496 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.759502 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.759556 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.759602 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260927 21:16:09.760354 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.755079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36259 (local address 127.0.0.1:56240)
0927 21:16:09.755421 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.755466 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.755536 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.755957 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.755958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.755959 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:09.755997 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.756003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.758399 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.758399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.759123 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.759125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.760132 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.760133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.760197 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.760201 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.760244 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.760285 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":85,"threads_started":1}
I20260927 21:16:09.760587 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.755186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.0.98.3:36259)
0927 21:16:09.755680 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.755684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.755695 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:16:09.755736 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.755738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.755753 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.755821 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:09.756079 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.757208 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.759213 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.759941 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.760013 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.760097 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.760468 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.760512 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":86,"threads_started":1}
I20260927 21:16:09.760504 18032 tablet_service.cc:1467] Processing CreateTablet for tablet 14c355853d8c48a8890e1168c9956b4d (DEFAULT_TABLE table=test-table [id=c48bfd7a93c847dcbff39c3d98a72ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:09.760627 18163 tablet_service.cc:1467] Processing CreateTablet for tablet 14c355853d8c48a8890e1168c9956b4d (DEFAULT_TABLE table=test-table [id=c48bfd7a93c847dcbff39c3d98a72ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:09.760897 18032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c355853d8c48a8890e1168c9956b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:09.760983 18163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c355853d8c48a8890e1168c9956b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:09.761302 18294 tablet_service.cc:1467] Processing CreateTablet for tablet 14c355853d8c48a8890e1168c9956b4d (DEFAULT_TABLE table=test-table [id=c48bfd7a93c847dcbff39c3d98a72ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:09.761649 18294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c355853d8c48a8890e1168c9956b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:09.763871 18374 tablet_bootstrap.cc:492] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777: Bootstrap starting.
I20260927 21:16:09.764328 18375 tablet_bootstrap.cc:492] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: Bootstrap starting.
I20260927 21:16:09.764580 18374 tablet_bootstrap.cc:654] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:09.764904 18374 log.cc:824] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:09.764953 18375 tablet_bootstrap.cc:654] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:09.765081 18376 tablet_bootstrap.cc:492] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: Bootstrap starting.
I20260927 21:16:09.765285 18375 log.cc:824] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:09.765770 18376 tablet_bootstrap.cc:654] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:09.765954 18374 tablet_bootstrap.cc:492] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777: No bootstrap required, opened a new log
I20260927 21:16:09.766021 18374 ts_tablet_manager.cc:1397] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:16:09.766081 18376 log.cc:824] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:09.766111 18375 tablet_bootstrap.cc:492] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: No bootstrap required, opened a new log
I20260927 21:16:09.766170 18375 ts_tablet_manager.cc:1397] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:16:09.766907 18376 tablet_bootstrap.cc:492] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: No bootstrap required, opened a new log
I20260927 21:16:09.766968 18376 ts_tablet_manager.cc:1397] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:16:09.768018 18374 raft_consensus.cc:348] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } }
I20260927 21:16:09.768154 18374 raft_consensus.cc:374] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:09.768174 18374 raft_consensus.cc:729] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231e22db7c35408c892ffd24244c0777, State: Initialized, Role: FOLLOWER
I20260927 21:16:09.768278 18374 consensus_queue.cc:260] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } }
I20260927 21:16:09.768539 18374 ts_tablet_manager.cc:1428] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:16:09.768576 18098 heartbeater.cc:503] Master 127.0.98.62:46343 was elected leader, sending a full tablet report...
I20260927 21:16:09.769450 18375 raft_consensus.cc:348] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } }
I20260927 21:16:09.769760 18375 raft_consensus.cc:374] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:09.769811 18375 raft_consensus.cc:729] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1594c5509a040db83e83824a28def2a, State: Initialized, Role: FOLLOWER
I20260927 21:16:09.769975 18375 consensus_queue.cc:260] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } }
I20260927 21:16:09.770277 18375 ts_tablet_manager.cc:1428] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:16:09.770183 18376 raft_consensus.cc:348] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } }
I20260927 21:16:09.770335 18376 raft_consensus.cc:374] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:09.770381 18376 raft_consensus.cc:729] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef64f71469bc4696b13f6e8b64edb33c, State: Initialized, Role: FOLLOWER
I20260927 21:16:09.770298 18229 heartbeater.cc:503] Master 127.0.98.62:46343 was elected leader, sending a full tablet report...
I20260927 21:16:09.770510 18376 consensus_queue.cc:260] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } }
I20260927 21:16:09.770756 18376 ts_tablet_manager.cc:1428] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:16:09.770824 18360 heartbeater.cc:503] Master 127.0.98.62:46343 was elected leader, sending a full tablet report...
W20260927 21:16:09.854816 18230 tablet.cc:2369] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:16:09.934742 18380 raft_consensus.cc:482] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:16:09.934898 18380 raft_consensus.cc:504] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } }
I20260927 21:16:09.935313 18380 leader_election.cc:288] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1594c5509a040db83e83824a28def2a (127.0.98.2:39069), ef64f71469bc4696b13f6e8b64edb33c (127.0.98.3:36259)
I20260927 21:16:09.938401 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.935505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39069 (local address 127.0.98.1:45101)
0927 21:16:09.935622 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.935633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.935685 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.935849 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.935850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.935852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:09.935942 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.935944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.936598 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.936599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.937496 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.937500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.938120 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.938121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.938201 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.938206 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.938266 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.938312 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 21:16:09.938427 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.935631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45101 (local address 127.0.98.2:39069)
0927 21:16:09.935678 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.935680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.935691 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:16:09.935749 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.935750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.935759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.935817 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:09.936021 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.936518 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.937588 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.938041 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.938096 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.938106 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.938302 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.938347 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:16:09.938550 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.935518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36259 (local address 127.0.98.1:49275)
0927 21:16:09.935665 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:09.935674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:09.935714 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:09.935898 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:09.935899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:09.935901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:09.935961 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.935963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.936736 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.936737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.937697 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:09.937700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.938301 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.938302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:09.938384 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.938388 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:09.938442 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:16:09.938480 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260927 21:16:09.938601 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.935719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49275 (local address 127.0.98.3:36259)
0927 21:16:09.935758 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.935759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.935766 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:16:09.935791 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.935792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.935800 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.935844 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:09.936046 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.936649 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.937789 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.938207 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.938273 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.938283 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.938466 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.938528 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:16:09.938922 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c355853d8c48a8890e1168c9956b4d" candidate_uuid: "231e22db7c35408c892ffd24244c0777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef64f71469bc4696b13f6e8b64edb33c" is_pre_election: true
I20260927 21:16:09.939103 18314 raft_consensus.cc:2454] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231e22db7c35408c892ffd24244c0777 in term 0.
I20260927 21:16:09.939363 17987 leader_election.cc:302] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231e22db7c35408c892ffd24244c0777, ef64f71469bc4696b13f6e8b64edb33c; no voters:
I20260927 21:16:09.939394 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c355853d8c48a8890e1168c9956b4d" candidate_uuid: "231e22db7c35408c892ffd24244c0777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1594c5509a040db83e83824a28def2a" is_pre_election: true
I20260927 21:16:09.939502 18380 raft_consensus.cc:2781] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:16:09.939532 18183 raft_consensus.cc:2454] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 231e22db7c35408c892ffd24244c0777 in term 0.
I20260927 21:16:09.939559 18380 raft_consensus.cc:482] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:16:09.939582 18380 raft_consensus.cc:3037] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:09.940318 18380 raft_consensus.cc:504] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } }
I20260927 21:16:09.940454 18380 leader_election.cc:288] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [CANDIDATE]: Term 1 election: Requested vote from peers a1594c5509a040db83e83824a28def2a (127.0.98.2:39069), ef64f71469bc4696b13f6e8b64edb33c (127.0.98.3:36259)
I20260927 21:16:09.940616 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c355853d8c48a8890e1168c9956b4d" candidate_uuid: "231e22db7c35408c892ffd24244c0777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef64f71469bc4696b13f6e8b64edb33c"
I20260927 21:16:09.940632 18183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c355853d8c48a8890e1168c9956b4d" candidate_uuid: "231e22db7c35408c892ffd24244c0777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1594c5509a040db83e83824a28def2a"
I20260927 21:16:09.940706 18183 raft_consensus.cc:3037] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:09.940706 18314 raft_consensus.cc:3037] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:09.941459 18183 raft_consensus.cc:2454] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231e22db7c35408c892ffd24244c0777 in term 1.
I20260927 21:16:09.941568 18314 raft_consensus.cc:2454] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 231e22db7c35408c892ffd24244c0777 in term 1.
I20260927 21:16:09.941669 17985 leader_election.cc:302] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231e22db7c35408c892ffd24244c0777, a1594c5509a040db83e83824a28def2a; no voters:
I20260927 21:16:09.941761 18380 raft_consensus.cc:2781] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:09.941900 18380 raft_consensus.cc:686] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [term 1 LEADER]: Becoming Leader. State: Replica: 231e22db7c35408c892ffd24244c0777, State: Running, Role: LEADER
I20260927 21:16:09.941982 18380 consensus_queue.cc:237] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } }
I20260927 21:16:09.942848 17910 catalog_manager.cc:5495] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 reported cstate change: term changed from 0 to 1, leader changed from <none> to 231e22db7c35408c892ffd24244c0777 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "231e22db7c35408c892ffd24244c0777" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "231e22db7c35408c892ffd24244c0777" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46381 } health_report { overall_health: HEALTHY } } }
W20260927 21:16:09.978555 18361 tablet.cc:2369] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:16:09.979096 18099 tablet.cc:2369] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:16:09.988051 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:09.985967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.0.98.1:46381)
0927 21:16:09.986024 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:09.986026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:09.986033 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:16:09.986072 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:09.986073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:09.986092 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:09.986150 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:09.986254 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.986729 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.987418 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:09.987620 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:09.987674 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:09.987774 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:09.987867 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:09.987917 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:09.987935 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 21:16:09.987954 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:16:09.989833 18183 raft_consensus.cc:1261] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [term 1 FOLLOWER]: Refusing update from remote peer 231e22db7c35408c892ffd24244c0777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:09.990114 18314 raft_consensus.cc:1261] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [term 1 FOLLOWER]: Refusing update from remote peer 231e22db7c35408c892ffd24244c0777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:09.990190 18380 consensus_queue.cc:1035] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1594c5509a040db83e83824a28def2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:09.990386 18384 consensus_queue.cc:1035] T 14c355853d8c48a8890e1168c9956b4d P 231e22db7c35408c892ffd24244c0777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef64f71469bc4696b13f6e8b64edb33c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:09.992765 18388 mvcc.cc:205] Tried to move back new op lower bound from 7334067281875677184 to 7334067281683030016. Current Snapshot: MvccSnapshot[applied={T|T < 7334067281875677184}]
I20260927 21:16:09.992873 18389 mvcc.cc:205] Tried to move back new op lower bound from 7334067281875677184 to 7334067281683030016. Current Snapshot: MvccSnapshot[applied={T|T < 7334067281875677184}]
I20260927 21:16:09.994858 17909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:09.994961 17909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:09.995754 17909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4b918865be34d0a9b5b667cd399f5c2: Sending DeleteTablet for 3 replicas of tablet 14c355853d8c48a8890e1168c9956b4d
I20260927 21:16:09.996125 18163 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c355853d8c48a8890e1168c9956b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:09 UTC) from {username='slave'} at 127.0.0.1:47846
I20260927 21:16:09.996126 18294 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c355853d8c48a8890e1168c9956b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:09 UTC) from {username='slave'} at 127.0.0.1:56240
I20260927 21:16:09.996459 18032 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c355853d8c48a8890e1168c9956b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:09 UTC) from {username='slave'} at 127.0.0.1:38666
I20260927 21:16:09.996665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 17970
I20260927 21:16:09.997604 18398 tablet_replica.cc:330] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: stopping tablet replica
I20260927 21:16:09.997699 18398 raft_consensus.cc:2229] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:16:09.997810 18398 raft_consensus.cc:2258] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:09.997977 18397 tablet_replica.cc:330] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: stopping tablet replica
I20260927 21:16:09.998080 18397 raft_consensus.cc:2229] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:16:09.998214 18397 raft_consensus.cc:2258] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:09.998282 18398 ts_tablet_manager.cc:1905] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:10.000159 18398 ts_tablet_manager.cc:1918] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:10.000241 18398 log.cc:1196] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/14c355853d8c48a8890e1168c9956b4d
I20260927 21:16:10.000531 18398 ts_tablet_manager.cc:1939] T 14c355853d8c48a8890e1168c9956b4d P ef64f71469bc4696b13f6e8b64edb33c: Deleting consensus metadata
I20260927 21:16:10.000563 18397 ts_tablet_manager.cc:1905] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:10.000998 17896 catalog_manager.cc:4841] TS ef64f71469bc4696b13f6e8b64edb33c (127.0.98.3:36259): tablet 14c355853d8c48a8890e1168c9956b4d (table test-table [id=c48bfd7a93c847dcbff39c3d98a72ddb]) successfully deleted
W20260927 21:16:10.001729 17895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46381 (error 108)
I20260927 21:16:10.002058 18397 ts_tablet_manager.cc:1918] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:10.002130 18397 log.cc:1196] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/14c355853d8c48a8890e1168c9956b4d
I20260927 21:16:10.002246 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:10.002094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46381 (local address 127.0.0.1:38686)
0927 21:16:10.002172 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:10.002211 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 21:16:10.002360 18397 ts_tablet_manager.cc:1939] T 14c355853d8c48a8890e1168c9956b4d P a1594c5509a040db83e83824a28def2a: Deleting consensus metadata
W20260927 21:16:10.002355 17895 catalog_manager.cc:4584] TS 231e22db7c35408c892ffd24244c0777 (127.0.98.1:46381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14c355853d8c48a8890e1168c9956b4d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46381: connect: Connection refused (error 111)
I20260927 21:16:10.002584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 18101
I20260927 21:16:10.007324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 18232
I20260927 21:16:10.007460 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:10.007328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39069 (local address 127.0.0.1:47860)
0927 21:16:10.007391 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:10.007425 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:16:10.011917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 17879
2026-09-27T21:16:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (837 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:10 dist-test-slave-r9x1 krb5kdc[18403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:10 dist-test-slave-r9x1 krb5kdc[18403](info): set up 2 sockets
Sep 27 21:16:10 dist-test-slave-r9x1 krb5kdc[18403](info): commencing operation
krb5kdc: starting...
W20260927 21:16:10.980822 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:11 dist-test-slave-r9x1 krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:11.105419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:11.186937 18416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:11.187168 18416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:11.187192 18416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:11.188450 18416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:11.188519 18416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:11.188532 18416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:11.188544 18416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:11.188553 18416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:11.190246 18416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:11.190508 18416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:11.190737 18416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:11.196105 18424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:11.196184 18416 server_base.cc:1034] running on GCE node
W20260927 21:16:11.196091 18422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:11.196089 18421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:11.196499 18416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:11.196811 18416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:11.197999 18416 hybrid_clock.cc:648] HybridClock initialized: now 1790543771197955 us; error 31 us; skew 500 ppm
Sep 27 21:16:11 dist-test-slave-r9x1 krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:11.200903 18416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:11.201436 18416 webserver.cc:460] Webserver started at http://127.0.98.62:33155/ using document root <none> and password file <none>
I20260927 21:16:11.201658 18416 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:11.201704 18416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:11.201830 18416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:11.202917 18416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "8ddb226da6964d28b0f6d13368a83bbc"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "a9b818ed48c3f25515f35090c415bb85"
server_key_iv: "0f0634caf96f2a81c4fbbe307c157562"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.203266 18416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ddb226da6964d28b0f6d13368a83bbc"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "a9b818ed48c3f25515f35090c415bb85"
server_key_iv: "0f0634caf96f2a81c4fbbe307c157562"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.204555 18416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:11.205277 18431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.205427 18416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:16:11.205518 18416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "8ddb226da6964d28b0f6d13368a83bbc"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "a9b818ed48c3f25515f35090c415bb85"
server_key_iv: "0f0634caf96f2a81c4fbbe307c157562"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.205590 18416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:11.230187 18416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:11.232133 18416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:11.232276 18416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:11.236063 18416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34727
I20260927 21:16:11.236097 18483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34727 every 8 connection(s)
I20260927 21:16:11.236445 18416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:11.237488 18484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:11.239923 18484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc: Bootstrap starting.
I20260927 21:16:11.240579 18484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:11.240631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 18416
I20260927 21:16:11.240752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:11.240914 18484 log.cc:824] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:11.240954 392 external_mini_cluster.cc:1428] Setting key 839232c762e9d87f3fd97abaee3f91af
I20260927 21:16:11.241959 18484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc: No bootstrap required, opened a new log
I20260927 21:16:11.244379 18484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddb226da6964d28b0f6d13368a83bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260927 21:16:11.244573 18484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:11.244608 18484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddb226da6964d28b0f6d13368a83bbc, State: Initialized, Role: FOLLOWER
I20260927 21:16:11.244743 18484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddb226da6964d28b0f6d13368a83bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260927 21:16:11.244827 18484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:11.244858 18484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:11.244910 18484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:11.245579 18484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddb226da6964d28b0f6d13368a83bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260927 21:16:11.245697 18484 leader_election.cc:302] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ddb226da6964d28b0f6d13368a83bbc; no voters:
I20260927 21:16:11.245872 18484 leader_election.cc:288] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:11.245966 18489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:11.246138 18489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [term 1 LEADER]: Becoming Leader. State: Replica: 8ddb226da6964d28b0f6d13368a83bbc, State: Running, Role: LEADER
I20260927 21:16:11.246155 18484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:11.246913 18489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddb226da6964d28b0f6d13368a83bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260927 21:16:11.247511 18490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ddb226da6964d28b0f6d13368a83bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddb226da6964d28b0f6d13368a83bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } } }
I20260927 21:16:11.247610 18490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:11.247520 18491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ddb226da6964d28b0f6d13368a83bbc. Latest consensus state: current_term: 1 leader_uuid: "8ddb226da6964d28b0f6d13368a83bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddb226da6964d28b0f6d13368a83bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } } }
I20260927 21:16:11.247748 18491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc [sys.catalog]: This master's current role is: LEADER
W20260927 21:16:11.247984 18503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:16:11.248059 18503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:11.248122 18504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:11.248586 18504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:16:11 dist-test-slave-r9x1 krb5kdc[18403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:11.252302 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.241494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.0.98.62:34727)
0927 21:16:11.241673 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.241682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.241718 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:16:11.242324 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.242334 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.242402 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.242576 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:11.243214 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.244635 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.245555 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.246184 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.246333 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.249140 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:11.249145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:11.249145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:11.249157 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:11.250885 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.251235 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.251237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.251238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.251318 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.251501 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.251502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.251503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.251735 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:11.251878 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.251999 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.252134 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260927 21:16:11.253391 18504 catalog_manager.cc:1347] Generated new cluster ID: 9805a5313d274810a6307b7e8828f89f
I20260927 21:16:11.253439 18504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:11.261112 18504 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:11.261667 18504 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:11.266407 18504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc: Generated new TSK 0
I20260927 21:16:11.266610 18504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:16:11.309221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34727
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:11.392652 18510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:11.392881 18510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:11.392910 18510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:11.394344 18510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:11.394403 18510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:11.394433 18510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:16:11.396193 18510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:11.396459 18510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:11.396696 18510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:11.402019 18516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:11.402038 18518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:11.402021 18515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:11.402307 18510 server_base.cc:1034] running on GCE node
I20260927 21:16:11.402555 18510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:11.402781 18510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:11.403956 18510 hybrid_clock.cc:648] HybridClock initialized: now 1790543771403939 us; error 32 us; skew 500 ppm
Sep 27 21:16:11 dist-test-slave-r9x1 krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:11.406807 18510 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:16:11.407361 18510 webserver.cc:460] Webserver started at http://127.0.98.1:40821/ using document root <none> and password file <none>
I20260927 21:16:11.407585 18510 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:11.407635 18510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:11.407766 18510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:11.408937 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "fb6f3658fc58482591059f158e129060"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "92f368af2e70ead1162b05d7fa3ef553"
server_key_iv: "e2ba6bbbaa0a1ee356898e4fe8ca0b15"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.409530 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fb6f3658fc58482591059f158e129060"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "92f368af2e70ead1162b05d7fa3ef553"
server_key_iv: "e2ba6bbbaa0a1ee356898e4fe8ca0b15"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.410948 18510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:11.411836 18525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.412133 18510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:11.412233 18510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "fb6f3658fc58482591059f158e129060"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "92f368af2e70ead1162b05d7fa3ef553"
server_key_iv: "e2ba6bbbaa0a1ee356898e4fe8ca0b15"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.412312 18510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:11.429999 18510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:11.432226 18510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:11.432367 18510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:11.432600 18510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:11.432906 18510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:11.432940 18510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.432969 18510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:11.433001 18510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.438782 18510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43381
I20260927 21:16:11.438895 18638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43381 every 8 connection(s)
I20260927 21:16:11.439169 18510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:16:11.444389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 18510
I20260927 21:16:11.444526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:16:11.444746 392 external_mini_cluster.cc:1428] Setting key b8d94285045ac0fb3c012ffdd014df79
Sep 27 21:16:11 dist-test-slave-r9x1 krb5kdc[18403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:11.449445 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.440042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49729 (local address 127.0.98.62:34727)
0927 21:16:11.440121 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.440124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.440220 (+ 96us) server_negotiation.cc:407] Connection header received
0927 21:16:11.441199 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.441200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.441211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.441287 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:11.441891 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.442408 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.443467 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.443665 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.443713 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.446280 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:11.446281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:11.446281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:11.446288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:11.447777 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.448279 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.448281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.448281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.448344 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.448650 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.448651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.448651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.448778 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:11.448843 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.449236 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.449369 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:16:11.449642 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.439925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.98.1:49729)
0927 21:16:11.440096 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:11.440139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:11.441084 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:11.441342 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:11.441345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:11.441614 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:11.441811 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.441815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.442500 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.442500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.443394 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.443397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.443764 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.443765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.443885 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.444627 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:11.444638 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:11.446169 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:11.447869 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:11.447877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:11.447880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:11.448178 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:11.448415 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:11.448416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:11.448416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:11.448560 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:11.448854 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:11.448861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:11.449043 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 21:16:11.449174 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 21:16:11.449388 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260927 21:16:11.450183 18639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34727
I20260927 21:16:11.450286 18639 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:11.450462 18639 heartbeater.cc:511] Master 127.0.98.62:34727 requested a full tablet report, sending...
I20260927 21:16:11.450964 18448 ts_manager.cc:194] Registered new tserver with Master: fb6f3658fc58482591059f158e129060 (127.0.98.1:43381)
I20260927 21:16:11.451586 18448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49729
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:16:11.473793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34727
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:11.555862 18644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:11.556109 18644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:11.556144 18644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:11.557473 18644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:11.557526 18644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:11.557564 18644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:16:11.559201 18644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:11.559465 18644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:11.559690 18644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:16:11.564718 18644 server_base.cc:1034] running on GCE node
W20260927 21:16:11.564683 18650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:11.564680 18652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:11.564683 18649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:11.565173 18644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:11.565476 18644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:11.566654 18644 hybrid_clock.cc:648] HybridClock initialized: now 1790543771566641 us; error 29 us; skew 500 ppm
Sep 27 21:16:11 dist-test-slave-r9x1 krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:11.569334 18644 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:16:11.569792 18644 webserver.cc:460] Webserver started at http://127.0.98.2:35485/ using document root <none> and password file <none>
I20260927 21:16:11.570000 18644 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:11.570056 18644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:11.570176 18644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:11.571249 18644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6ef11a7ff614095afa75b60fe71b03f"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "14756189d14a28bbc83687368fc8333d"
server_key_iv: "8ff9b301834e581d8c6b98d7222f5952"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.571591 18644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6ef11a7ff614095afa75b60fe71b03f"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "14756189d14a28bbc83687368fc8333d"
server_key_iv: "8ff9b301834e581d8c6b98d7222f5952"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.573045 18644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:11.573971 18659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.574169 18644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:16:11.574254 18644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "c6ef11a7ff614095afa75b60fe71b03f"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "14756189d14a28bbc83687368fc8333d"
server_key_iv: "8ff9b301834e581d8c6b98d7222f5952"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.574333 18644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:11.587389 18644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:11.589658 18644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:11.589778 18644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:11.589998 18644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:11.590315 18644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:11.590348 18644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.590378 18644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:11.590396 18644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.595892 18644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39531
I20260927 21:16:11.595932 18772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39531 every 8 connection(s)
I20260927 21:16:11.596280 18644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:16:11.598836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 18644
I20260927 21:16:11.598945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:16:11.599149 392 external_mini_cluster.cc:1428] Setting key 3e5f4ba3fb600291e21cad1ca5e21917
Sep 27 21:16:11 dist-test-slave-r9x1 krb5kdc[18403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:11.607061 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.597032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40091 (local address 127.0.98.62:34727)
0927 21:16:11.597097 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.597099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.597283 (+ 184us) server_negotiation.cc:407] Connection header received
0927 21:16:11.598258 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.598260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.598273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.598353 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:11.599143 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.599733 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.601168 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.601395 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.601485 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.604221 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:11.604223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:11.604223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:11.604230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:11.605586 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.606091 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.606092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.606093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.606143 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.606409 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.606410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.606410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.606522 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:11.606568 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.606917 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.606981 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:16:11.607327 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.596972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.98.2:40091)
0927 21:16:11.597140 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:11.597183 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:11.598120 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:11.598412 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:11.598416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:11.598717 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:11.599034 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.599041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.599827 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.599828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.601045 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.601049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.601498 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.601500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.601756 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.602292 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:11.602305 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:11.604092 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:11.605675 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:11.605682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:11.605685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:11.606013 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:11.606207 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:11.606207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:11.606208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:11.606349 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:11.606586 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:11.606592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:11.606771 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 21:16:11.606905 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 21:16:11.607083 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:16:11.607909 18773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34727
I20260927 21:16:11.608021 18773 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:11.608224 18773 heartbeater.cc:511] Master 127.0.98.62:34727 requested a full tablet report, sending...
I20260927 21:16:11.608641 18448 ts_manager.cc:194] Registered new tserver with Master: c6ef11a7ff614095afa75b60fe71b03f (127.0.98.2:39531)
I20260927 21:16:11.609112 18448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:16:11.629149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34727
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:11.712901 18778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:11.713160 18778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:11.713188 18778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:11.714556 18778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:11.714607 18778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:11.714637 18778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:16:11.716320 18778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:11.716566 18778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:11.716787 18778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:11.722034 18783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:11.722059 18784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:11.722067 18786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:11.722363 18778 server_base.cc:1034] running on GCE node
I20260927 21:16:11.722710 18778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:11.723029 18778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:11.724309 18778 hybrid_clock.cc:648] HybridClock initialized: now 1790543771724292 us; error 32 us; skew 500 ppm
Sep 27 21:16:11 dist-test-slave-r9x1 krb5kdc[18403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:11.727203 18778 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:16:11.727658 18778 webserver.cc:460] Webserver started at http://127.0.98.3:41587/ using document root <none> and password file <none>
I20260927 21:16:11.727850 18778 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:11.727891 18778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:11.727981 18778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:11.728999 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "3cb353175b7e4de1ae45e83a200e13a8"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "4b7f5866dfe6ee4db5cfefd6f1f3d4f7"
server_key_iv: "357316cc76220472ce1f41d630e2eadb"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.729349 18778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3cb353175b7e4de1ae45e83a200e13a8"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "4b7f5866dfe6ee4db5cfefd6f1f3d4f7"
server_key_iv: "357316cc76220472ce1f41d630e2eadb"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.730701 18778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:11.731528 18793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.731748 18778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:11.731860 18778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "3cb353175b7e4de1ae45e83a200e13a8"
format_stamp: "Formatted at 2026-09-27 21:16:11 on dist-test-slave-r9x1"
server_key: "4b7f5866dfe6ee4db5cfefd6f1f3d4f7"
server_key_iv: "357316cc76220472ce1f41d630e2eadb"
server_key_version: "encryptionkey@0"
I20260927 21:16:11.731930 18778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:11.750818 18778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:11.753398 18778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:11.753551 18778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:11.753773 18778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:11.754092 18778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:11.754132 18778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.754161 18778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:11.754177 18778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:11.759771 18778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37443
I20260927 21:16:11.759831 18906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37443 every 8 connection(s)
I20260927 21:16:11.760146 18778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:16:11.764555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 18778
I20260927 21:16:11.764649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:16:11.764827 392 external_mini_cluster.cc:1428] Setting key 6155724cf5ccc4679fe5c5fcdbd9fedd
Sep 27 21:16:11 dist-test-slave-r9x1 krb5kdc[18403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:11.770637 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.761154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49013 (local address 127.0.98.62:34727)
0927 21:16:11.761213 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.761215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.761434 (+ 219us) server_negotiation.cc:407] Connection header received
0927 21:16:11.762433 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.762434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.762445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.762512 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:11.763210 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.763737 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.765231 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.765531 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.765659 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.767850 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:11.767852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:11.767852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:11.767860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:11.769253 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.769701 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.769702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.769702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.769759 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.770033 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.770035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.770035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.770200 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:11.770257 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.770497 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.770565 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:16:11.770823 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.761080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34727 (local address 127.0.98.3:49013)
0927 21:16:11.761315 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:11.761360 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:11.762316 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:11.762581 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:11.762584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:11.762874 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:11.763097 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.763101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.763854 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.763855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.765172 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.765176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.765669 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.765670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.765807 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.766333 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:11.766341 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:11.767752 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:11.769369 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:11.769376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:11.769378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:11.769625 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:11.769815 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:11.769815 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:11.769816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:11.769946 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:11.770251 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:11.770256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:11.770386 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 21:16:11.770471 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:16:11.770603 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260927 21:16:11.771369 18907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34727
I20260927 21:16:11.771473 18907 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:11.771656 18907 heartbeater.cc:511] Master 127.0.98.62:34727 requested a full tablet report, sending...
I20260927 21:16:11.772023 18448 ts_manager.cc:194] Registered new tserver with Master: 3cb353175b7e4de1ae45e83a200e13a8 (127.0.98.3:37443)
I20260927 21:16:11.772382 18448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49013
I20260927 21:16:11.777110 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:16:11.784818 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.778442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.0.98.62:34727)
0927 21:16:11.778551 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.778553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.778583 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:16:11.778665 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.778665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.778675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.778721 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:11.779309 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.779810 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.780709 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.780908 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.780965 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.782339 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:11.782340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:11.782341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:11.782348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:11.783567 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.783990 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.783991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.783991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.784044 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:11.784251 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:11.784252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:11.784252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:11.784405 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:11.784508 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.784650 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.784719 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:16:11.786556 18448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:16:11.788172 18448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:16:11.795167 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.790574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43381 (local address 127.0.0.1:50620)
0927 21:16:11.790733 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:11.790758 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:11.790824 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:11.791286 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:11.791287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:11.791291 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:11.791398 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.791401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.792795 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.792796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.793718 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.793721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.794807 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.794809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.794899 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.794915 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 21:16:11.794999 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:16:11.795064 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260927 21:16:11.795251 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.790955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.0.98.1:43381)
0927 21:16:11.791108 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.791114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.791125 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:16:11.791165 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.791167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.791183 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.791478 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:11.791555 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.792657 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.793797 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.794706 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.794791 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.794858 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.795031 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.795079 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260927 21:16:11.795274 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.790771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37443 (local address 127.0.0.1:45078)
0927 21:16:11.790968 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:11.790977 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:11.791018 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:11.791282 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:11.791285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:11.791287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:11.791378 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.791381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.792363 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.792364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.793735 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.793738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.794957 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.794958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.795022 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.795026 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:11.795077 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:16:11.795202 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":59,"threads_started":1}
I20260927 21:16:11.795357 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.790569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39531 (local address 127.0.0.1:46996)
0927 21:16:11.790714 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:11.790757 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:11.790824 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:11.791286 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:11.791287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:11.791289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:11.791398 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.791401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.792759 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.792760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.793831 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.793833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.795116 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.795117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.795184 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.795188 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:11.795215 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 21:16:11.795251 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":6,"thread_start_us":69,"threads_started":1}
I20260927 21:16:11.795516 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.790957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.0.98.3:37443)
0927 21:16:11.791137 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.791140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.791147 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:16:11.791177 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.791178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.791190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.791252 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:11.791511 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.792257 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.793821 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.794706 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.794791 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.794905 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.795388 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.795449 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
I20260927 21:16:11.796078 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.790724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.0.98.2:39531)
0927 21:16:11.791132 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.791136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.791145 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:11.791178 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.791180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.791194 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.791477 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:11.791556 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.792658 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.794379 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.795027 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.795851 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.795936 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.795963 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.796015 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":76,"threads_started":1}
I20260927 21:16:11.796696 18707 tablet_service.cc:1467] Processing CreateTablet for tablet ed4af1ea6ad643debb0f4cc29b8b0df9 (DEFAULT_TABLE table=test-table [id=2a28d4876edd4460b031d26ba405b944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:11.796904 18573 tablet_service.cc:1467] Processing CreateTablet for tablet ed4af1ea6ad643debb0f4cc29b8b0df9 (DEFAULT_TABLE table=test-table [id=2a28d4876edd4460b031d26ba405b944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:11.797151 18707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4af1ea6ad643debb0f4cc29b8b0df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:11.797240 18573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4af1ea6ad643debb0f4cc29b8b0df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:11.797927 18841 tablet_service.cc:1467] Processing CreateTablet for tablet ed4af1ea6ad643debb0f4cc29b8b0df9 (DEFAULT_TABLE table=test-table [id=2a28d4876edd4460b031d26ba405b944]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:11.798274 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed4af1ea6ad643debb0f4cc29b8b0df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:11.800037 18924 tablet_bootstrap.cc:492] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: Bootstrap starting.
I20260927 21:16:11.800786 18924 tablet_bootstrap.cc:654] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:11.801076 18925 tablet_bootstrap.cc:492] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060: Bootstrap starting.
I20260927 21:16:11.801383 18924 log.cc:824] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:11.801767 18925 tablet_bootstrap.cc:654] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:11.802037 18927 tablet_bootstrap.cc:492] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: Bootstrap starting.
I20260927 21:16:11.802263 18924 tablet_bootstrap.cc:492] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: No bootstrap required, opened a new log
I20260927 21:16:11.802305 18925 log.cc:824] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:11.802341 18924 ts_tablet_manager.cc:1397] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:16:11.802651 18927 tablet_bootstrap.cc:654] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:11.803112 18925 tablet_bootstrap.cc:492] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060: No bootstrap required, opened a new log
I20260927 21:16:11.803191 18925 ts_tablet_manager.cc:1397] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:16:11.803244 18927 log.cc:824] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:11.804059 18927 tablet_bootstrap.cc:492] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: No bootstrap required, opened a new log
I20260927 21:16:11.804128 18927 ts_tablet_manager.cc:1397] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:16:11.804231 18924 raft_consensus.cc:348] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } }
I20260927 21:16:11.804538 18924 raft_consensus.cc:374] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:11.804621 18924 raft_consensus.cc:729] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ef11a7ff614095afa75b60fe71b03f, State: Initialized, Role: FOLLOWER
I20260927 21:16:11.804757 18924 consensus_queue.cc:260] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } }
I20260927 21:16:11.805132 18773 heartbeater.cc:503] Master 127.0.98.62:34727 was elected leader, sending a full tablet report...
I20260927 21:16:11.805176 18925 raft_consensus.cc:348] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } }
I20260927 21:16:11.805368 18925 raft_consensus.cc:374] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:11.805135 18924 ts_tablet_manager.cc:1428] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:16:11.805410 18925 raft_consensus.cc:729] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb6f3658fc58482591059f158e129060, State: Initialized, Role: FOLLOWER
I20260927 21:16:11.805554 18925 consensus_queue.cc:260] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } }
I20260927 21:16:11.805852 18639 heartbeater.cc:503] Master 127.0.98.62:34727 was elected leader, sending a full tablet report...
I20260927 21:16:11.805841 18925 ts_tablet_manager.cc:1428] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:16:11.806561 18927 raft_consensus.cc:348] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } }
I20260927 21:16:11.806733 18927 raft_consensus.cc:374] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:11.806777 18927 raft_consensus.cc:729] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb353175b7e4de1ae45e83a200e13a8, State: Initialized, Role: FOLLOWER
I20260927 21:16:11.806885 18927 consensus_queue.cc:260] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } }
I20260927 21:16:11.807153 18927 ts_tablet_manager.cc:1428] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:16:11.807178 18907 heartbeater.cc:503] Master 127.0.98.62:34727 was elected leader, sending a full tablet report...
W20260927 21:16:11.846824 18774 tablet.cc:2369] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:16:11.899540 18930 raft_consensus.cc:482] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:16:11.899679 18930 raft_consensus.cc:504] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } }
I20260927 21:16:11.900048 18930 leader_election.cc:288] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb6f3658fc58482591059f158e129060 (127.0.98.1:43381), 3cb353175b7e4de1ae45e83a200e13a8 (127.0.98.3:37443)
I20260927 21:16:11.902997 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.900389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37941 (local address 127.0.98.1:43381)
0927 21:16:11.900457 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.900460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.900473 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:16:11.900508 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.900509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.900521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.900602 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:11.900764 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.901239 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.902266 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.902643 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.902704 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.902716 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.902887 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.902930 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:16:11.902997 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.900308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43381 (local address 127.0.98.2:37941)
0927 21:16:11.900389 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:11.900402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:11.900450 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:11.900610 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:11.900611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:11.900614 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:11.900690 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.900692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.901354 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.901355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.902157 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.902160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.902730 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.902731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.902798 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.902801 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:11.902868 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:16:11.902913 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":3}
I20260927 21:16:11.903060 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.900503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43649 (local address 127.0.98.3:37443)
0927 21:16:11.900556 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.900558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.900662 (+ 104us) server_negotiation.cc:407] Connection header received
0927 21:16:11.900701 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.900702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.900711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.900750 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:11.900912 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.901406 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.902345 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.902711 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.902769 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.902779 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.902959 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.902999 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:16:11.903121 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.900419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37443 (local address 127.0.98.2:43649)
0927 21:16:11.900595 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:11.900602 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:11.900650 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:11.900770 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:11.900770 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:11.900772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:11.900847 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.900848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.901479 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.901480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.902242 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:11.902244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.902795 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.902796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:11.902888 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.902892 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:11.902939 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:16:11.902979 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":110,"threads_started":1}
I20260927 21:16:11.903414 18593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4af1ea6ad643debb0f4cc29b8b0df9" candidate_uuid: "c6ef11a7ff614095afa75b60fe71b03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6f3658fc58482591059f158e129060" is_pre_election: true
I20260927 21:16:11.903425 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4af1ea6ad643debb0f4cc29b8b0df9" candidate_uuid: "c6ef11a7ff614095afa75b60fe71b03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb353175b7e4de1ae45e83a200e13a8" is_pre_election: true
I20260927 21:16:11.903586 18861 raft_consensus.cc:2454] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ef11a7ff614095afa75b60fe71b03f in term 0.
I20260927 21:16:11.903587 18593 raft_consensus.cc:2454] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ef11a7ff614095afa75b60fe71b03f in term 0.
I20260927 21:16:11.903887 18663 leader_election.cc:302] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb353175b7e4de1ae45e83a200e13a8, c6ef11a7ff614095afa75b60fe71b03f; no voters:
I20260927 21:16:11.904065 18930 raft_consensus.cc:2781] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:16:11.904142 18930 raft_consensus.cc:482] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:16:11.904168 18930 raft_consensus.cc:3037] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:11.905195 18930 raft_consensus.cc:504] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } }
I20260927 21:16:11.905387 18930 leader_election.cc:288] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [CANDIDATE]: Term 1 election: Requested vote from peers fb6f3658fc58482591059f158e129060 (127.0.98.1:43381), 3cb353175b7e4de1ae45e83a200e13a8 (127.0.98.3:37443)
I20260927 21:16:11.905624 18593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4af1ea6ad643debb0f4cc29b8b0df9" candidate_uuid: "c6ef11a7ff614095afa75b60fe71b03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb6f3658fc58482591059f158e129060"
I20260927 21:16:11.905678 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed4af1ea6ad643debb0f4cc29b8b0df9" candidate_uuid: "c6ef11a7ff614095afa75b60fe71b03f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cb353175b7e4de1ae45e83a200e13a8"
I20260927 21:16:11.905714 18593 raft_consensus.cc:3037] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:11.905777 18861 raft_consensus.cc:3037] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:11.906635 18593 raft_consensus.cc:2454] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ef11a7ff614095afa75b60fe71b03f in term 1.
I20260927 21:16:11.906720 18861 raft_consensus.cc:2454] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ef11a7ff614095afa75b60fe71b03f in term 1.
I20260927 21:16:11.906867 18663 leader_election.cc:302] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ef11a7ff614095afa75b60fe71b03f, fb6f3658fc58482591059f158e129060; no voters:
I20260927 21:16:11.907006 18930 raft_consensus.cc:2781] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:11.907176 18930 raft_consensus.cc:686] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 1 LEADER]: Becoming Leader. State: Replica: c6ef11a7ff614095afa75b60fe71b03f, State: Running, Role: LEADER
I20260927 21:16:11.907317 18930 consensus_queue.cc:237] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } }
I20260927 21:16:11.908334 18448 catalog_manager.cc:5495] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ef11a7ff614095afa75b60fe71b03f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c6ef11a7ff614095afa75b60fe71b03f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ef11a7ff614095afa75b60fe71b03f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 } health_report { overall_health: UNKNOWN } } }
I20260927 21:16:11.914217 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:11.911635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.0.98.2:39531)
0927 21:16:11.911685 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:11.911687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:11.911748 (+ 61us) server_negotiation.cc:407] Connection header received
0927 21:16:11.911839 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:11.911840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:11.911864 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:11.911924 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:11.912077 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.912557 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.913342 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:11.913556 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:11.913613 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:11.913783 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:11.913886 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:11.914007 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:11.914036 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 21:16:11.914048 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:16:11.916251 18861 raft_consensus.cc:1261] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [term 1 FOLLOWER]: Refusing update from remote peer c6ef11a7ff614095afa75b60fe71b03f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:11.916610 18593 raft_consensus.cc:1261] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [term 1 FOLLOWER]: Refusing update from remote peer c6ef11a7ff614095afa75b60fe71b03f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:11.916618 18930 consensus_queue.cc:1035] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cb353175b7e4de1ae45e83a200e13a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:11.916913 18930 consensus_queue.cc:1035] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6f3658fc58482591059f158e129060" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:11.919764 18938 mvcc.cc:205] Tried to move back new op lower bound from 7334067289765908480 to 7334067289733140480. Current Snapshot: MvccSnapshot[applied={T|T < 7334067289765908480}]
I20260927 21:16:11.920362 18939 mvcc.cc:205] Tried to move back new op lower bound from 7334067289765908480 to 7334067289733140480. Current Snapshot: MvccSnapshot[applied={T|T < 7334067289765908480}]
I20260927 21:16:11.921782 18447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:11.921895 18447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:11.922797 18447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ddb226da6964d28b0f6d13368a83bbc: Sending DeleteTablet for 3 replicas of tablet ed4af1ea6ad643debb0f4cc29b8b0df9
I20260927 21:16:11.923180 18707 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4af1ea6ad643debb0f4cc29b8b0df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46996
I20260927 21:16:11.923277 18573 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4af1ea6ad643debb0f4cc29b8b0df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50620
I20260927 21:16:11.923523 18948 tablet_replica.cc:330] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060: stopping tablet replica
I20260927 21:16:11.923527 18841 tablet_service.cc:1514] Processing DeleteTablet for tablet ed4af1ea6ad643debb0f4cc29b8b0df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45078
I20260927 21:16:11.923645 18948 raft_consensus.cc:2229] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:16:11.923772 18948 raft_consensus.cc:2258] T ed4af1ea6ad643debb0f4cc29b8b0df9 P fb6f3658fc58482591059f158e129060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:11.923777 18949 tablet_replica.cc:330] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: stopping tablet replica
I20260927 21:16:11.923848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 18510
I20260927 21:16:11.923986 18949 raft_consensus.cc:2229] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:16:11.924144 18949 raft_consensus.cc:2258] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:11.924643 18949 ts_tablet_manager.cc:1905] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:11.926545 18949 ts_tablet_manager.cc:1918] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:11.926633 18949 log.cc:1196] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/ed4af1ea6ad643debb0f4cc29b8b0df9
I20260927 21:16:11.926872 18949 ts_tablet_manager.cc:1939] T ed4af1ea6ad643debb0f4cc29b8b0df9 P 3cb353175b7e4de1ae45e83a200e13a8: Deleting consensus metadata
I20260927 21:16:11.927359 18435 catalog_manager.cc:4841] TS 3cb353175b7e4de1ae45e83a200e13a8 (127.0.98.3:37443): tablet ed4af1ea6ad643debb0f4cc29b8b0df9 (table test-table [id=2a28d4876edd4460b031d26ba405b944]) successfully deleted
I20260927 21:16:11.929430 18947 tablet_replica.cc:330] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: stopping tablet replica
I20260927 21:16:11.929586 18947 raft_consensus.cc:2229] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:16:11.929713 18947 raft_consensus.cc:2258] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:11.930124 18947 ts_tablet_manager.cc:1905] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:11.931836 18947 ts_tablet_manager.cc:1918] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:11.931918 18947 log.cc:1196] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/ed4af1ea6ad643debb0f4cc29b8b0df9
I20260927 21:16:11.932186 18947 ts_tablet_manager.cc:1939] T ed4af1ea6ad643debb0f4cc29b8b0df9 P c6ef11a7ff614095afa75b60fe71b03f: Deleting consensus metadata
W20260927 21:16:11.932214 18435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43381 (error 108)
I20260927 21:16:11.932623 18433 catalog_manager.cc:4841] TS c6ef11a7ff614095afa75b60fe71b03f (127.0.98.2:39531): tablet ed4af1ea6ad643debb0f4cc29b8b0df9 (table test-table [id=2a28d4876edd4460b031d26ba405b944]) successfully deleted
W20260927 21:16:11.932802 18919 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:11.932657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43381 (local address 127.0.0.1:50626)
0927 21:16:11.932720 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:11.932764 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43381: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 21:16:11.932916 18435 catalog_manager.cc:4584] TS fb6f3658fc58482591059f158e129060 (127.0.98.1:43381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed4af1ea6ad643debb0f4cc29b8b0df9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43381: connect: Connection reset by peer (error 104)
I20260927 21:16:11.933040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 18644
I20260927 21:16:11.938315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 18778
I20260927 21:16:11.943069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 18416
2026-09-27T21:16:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1932 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T21:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:11Z Disabled control of system clock
I20260927 21:16:11.972829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42047
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:16:12.058553 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:12.058791 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:12.058820 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:12.060065 18954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:12.060122 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:12.060137 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:12.060153 18954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:12.060168 18954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:12.061645 18954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:12.061914 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:12.062203 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:12.067128 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:12.067176 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:12.067166 18954 server_base.cc:1034] running on GCE node
W20260927 21:16:12.067122 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:12.067958 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:12.068257 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:12.069391 18954 hybrid_clock.cc:648] HybridClock initialized: now 1790543772069370 us; error 40 us; skew 500 ppm
I20260927 21:16:12.070647 18954 webserver.cc:460] Webserver started at http://127.0.98.62:45013/ using document root <none> and password file <none>
I20260927 21:16:12.070878 18954 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:12.070925 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:12.071067 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:12.072177 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "4e5001c8d6d240a3a2ee5f576a2072a3"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "75746b1ce615a28d5ba074252920714d"
server_key_iv: "9a6f20be7ebf85f664d6e3479690d9ec"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.072522 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e5001c8d6d240a3a2ee5f576a2072a3"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "75746b1ce615a28d5ba074252920714d"
server_key_iv: "9a6f20be7ebf85f664d6e3479690d9ec"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.073943 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:12.074909 18968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.075085 18954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:12.075177 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "4e5001c8d6d240a3a2ee5f576a2072a3"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "75746b1ce615a28d5ba074252920714d"
server_key_iv: "9a6f20be7ebf85f664d6e3479690d9ec"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.075254 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:12.091238 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:12.091595 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:12.091719 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:12.095471 18954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42047
I20260927 21:16:12.095505 19020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42047 every 8 connection(s)
I20260927 21:16:12.095853 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:12.096872 19021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:12.097298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 18954
I20260927 21:16:12.097397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:12.097615 392 external_mini_cluster.cc:1428] Setting key 5f5e4136cc3f88a7718a5e0f030a5b67
I20260927 21:16:12.099663 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3: Bootstrap starting.
I20260927 21:16:12.100256 19021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:12.100529 19021 log.cc:824] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:12.101197 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3: No bootstrap required, opened a new log
I20260927 21:16:12.103339 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.098187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.0.98.62:42047)
0927 21:16:12.098395 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.098403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.098429 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:16:12.098972 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.098980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.099046 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.099185 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:12.099436 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.100356 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.101301 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.101735 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.101850 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.102683 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:12.102687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:12.102687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:12.102870 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:12.102904 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:12.102916 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:12.102970 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.103092 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.103177 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260927 21:16:12.103263 19021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5001c8d6d240a3a2ee5f576a2072a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } }
I20260927 21:16:12.103428 19021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:12.103467 19021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5001c8d6d240a3a2ee5f576a2072a3, State: Initialized, Role: FOLLOWER
I20260927 21:16:12.103605 19021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5001c8d6d240a3a2ee5f576a2072a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } }
I20260927 21:16:12.103677 19021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:12.103706 19021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:12.103752 19021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:12.104543 19021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5001c8d6d240a3a2ee5f576a2072a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } }
I20260927 21:16:12.104645 19021 leader_election.cc:302] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e5001c8d6d240a3a2ee5f576a2072a3; no voters:
I20260927 21:16:12.104820 19021 leader_election.cc:288] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:12.104897 19026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:12.105047 19026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [term 1 LEADER]: Becoming Leader. State: Replica: 4e5001c8d6d240a3a2ee5f576a2072a3, State: Running, Role: LEADER
I20260927 21:16:12.105089 19021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:12.105140 19026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5001c8d6d240a3a2ee5f576a2072a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } }
I20260927 21:16:12.105636 19028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e5001c8d6d240a3a2ee5f576a2072a3. Latest consensus state: current_term: 1 leader_uuid: "4e5001c8d6d240a3a2ee5f576a2072a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5001c8d6d240a3a2ee5f576a2072a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } } }
I20260927 21:16:12.105733 19028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:12.105930 19035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:12.106096 19027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e5001c8d6d240a3a2ee5f576a2072a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e5001c8d6d240a3a2ee5f576a2072a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42047 } } }
I20260927 21:16:12.106251 19027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:12.106662 19035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:12.108259 19035 catalog_manager.cc:1347] Generated new cluster ID: 26698cd17c314ebd896305a709e14369
I20260927 21:16:12.108318 19035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:12.116581 19035 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:12.117139 19035 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:12.128340 19035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3: Generated new TSK 0
I20260927 21:16:12.128566 19035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:12.132444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:16:12.216250 19045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:12.216482 19045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:12.216504 19045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:12.217828 19045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:12.217880 19045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:12.217907 19045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:16:12.219337 19045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:12.219597 19045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:12.219909 19045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:12.224890 19053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:12.224941 19051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:12.225164 19045 server_base.cc:1034] running on GCE node
W20260927 21:16:12.224890 19050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:12.225467 19045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:12.225723 19045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:12.226886 19045 hybrid_clock.cc:648] HybridClock initialized: now 1790543772226866 us; error 33 us; skew 500 ppm
I20260927 21:16:12.228113 19045 webserver.cc:460] Webserver started at http://127.0.98.1:36415/ using document root <none> and password file <none>
I20260927 21:16:12.228374 19045 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:12.228420 19045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:12.228545 19045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:12.229671 19045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "c4196e0939fd421c8f45589ed3ee7c94"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "c8655b8ac5227269876da82322957240"
server_key_iv: "77b39bba89c4e362b569768cb18fa45d"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.229992 19045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c4196e0939fd421c8f45589ed3ee7c94"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "c8655b8ac5227269876da82322957240"
server_key_iv: "77b39bba89c4e362b569768cb18fa45d"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.231233 19045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:16:12.231997 19059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.232229 19045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:12.232318 19045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "c4196e0939fd421c8f45589ed3ee7c94"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "c8655b8ac5227269876da82322957240"
server_key_iv: "77b39bba89c4e362b569768cb18fa45d"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.232398 19045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:12.261021 19045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:12.261396 19045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:12.261545 19045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:12.261785 19045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:12.262112 19045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:12.262147 19045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.262177 19045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:12.262195 19045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.267724 19045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36471
I20260927 21:16:12.267795 19172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36471 every 8 connection(s)
I20260927 21:16:12.268115 19045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:16:12.276762 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.268888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51465 (local address 127.0.98.62:42047)
0927 21:16:12.268959 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.268961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.269141 (+ 180us) server_negotiation.cc:407] Connection header received
0927 21:16:12.269671 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.269671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.269685 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.269740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:12.270397 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.270911 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.271874 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.272077 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.272132 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.276452 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:12.276454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:12.276454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:12.276461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:12.276478 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:12.276484 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:12.276542 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.276653 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.276691 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:16:12.276997 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.268838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42047 (local address 127.0.98.1:51465)
0927 21:16:12.269020 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:12.269057 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:12.269584 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:12.269793 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:12.269796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:12.269804 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:12.270319 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.270323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.270989 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.270989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.271794 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.271797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.272161 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.272162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.272279 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.276221 (+ 3942us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:16:12.276222 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:12.276265 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:16:12.276297 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:12.276569 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:12.276575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:12.276579 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:12.276643 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:16:12.276755 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260927 21:16:12.277622 19173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42047
I20260927 21:16:12.277733 19173 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:12.277791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 19045
I20260927 21:16:12.277889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:16:12.277951 19173 heartbeater.cc:511] Master 127.0.98.62:42047 requested a full tablet report, sending...
I20260927 21:16:12.278153 392 external_mini_cluster.cc:1428] Setting key e24f71a0ef085843ad47820908bf586a
I20260927 21:16:12.278628 18985 ts_manager.cc:194] Registered new tserver with Master: c4196e0939fd421c8f45589ed3ee7c94 (127.0.98.1:36471)
I20260927 21:16:12.279124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:16:12.279345 18985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51465
W20260927 21:16:12.360092 19176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:12.360325 19176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:12.360349 19176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:12.361585 19176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:12.361625 19176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:12.361651 19176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:16:12.363080 19176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:12.363302 19176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:12.363619 19176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:12.368918 19182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:12.368920 19181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:12.368971 19176 server_base.cc:1034] running on GCE node
W20260927 21:16:12.368920 19184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:12.369504 19176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:12.369765 19176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:12.370954 19176 hybrid_clock.cc:648] HybridClock initialized: now 1790543772370936 us; error 32 us; skew 500 ppm
I20260927 21:16:12.372380 19176 webserver.cc:460] Webserver started at http://127.0.98.2:35599/ using document root <none> and password file <none>
I20260927 21:16:12.372718 19176 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:12.372797 19176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:12.372941 19176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:12.374127 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "619561fcaed44c28aef3387f598df225"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "967a6745690615419c13d269460cfd26"
server_key_iv: "7b33853adecc76f9431a46556b1d0b43"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.374475 19176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "619561fcaed44c28aef3387f598df225"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "967a6745690615419c13d269460cfd26"
server_key_iv: "7b33853adecc76f9431a46556b1d0b43"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.375800 19176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:16:12.376608 19190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.376807 19176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:12.376893 19176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "619561fcaed44c28aef3387f598df225"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "967a6745690615419c13d269460cfd26"
server_key_iv: "7b33853adecc76f9431a46556b1d0b43"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.376983 19176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:12.405174 19176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:12.405539 19176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:12.405678 19176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:12.405916 19176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:12.406253 19176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:12.406288 19176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.406317 19176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:12.406334 19176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.412626 19176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43371
I20260927 21:16:12.412676 19303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43371 every 8 connection(s)
I20260927 21:16:12.413015 19176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:16:12.414362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 19176
I20260927 21:16:12.414451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:16:12.414641 392 external_mini_cluster.cc:1428] Setting key bc504d6f432c3f6bb639f8436c26d70c
I20260927 21:16:12.415558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:16:12.418509 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.413849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53407 (local address 127.0.98.62:42047)
0927 21:16:12.413915 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.413951 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.414142 (+ 191us) server_negotiation.cc:407] Connection header received
0927 21:16:12.414742 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.414744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.414757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.414810 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:12.415512 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.416059 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.417046 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.417237 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.417335 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.418182 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:12.418183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:12.418183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:12.418188 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:12.418204 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:12.418208 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:12.418253 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.418381 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.418414 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:16:12.418651 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.413814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42047 (local address 127.0.98.2:53407)
0927 21:16:12.414020 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:12.414058 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:12.414615 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:12.414905 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:12.414910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:12.414920 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:12.415410 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.415414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.416153 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.416154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.416954 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.416957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.417347 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.417348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.417498 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.418034 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:16:12.418034 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:12.418066 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:16:12.418092 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:12.418263 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:12.418269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:12.418272 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:12.418343 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:16:12.418457 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260927 21:16:12.419130 19304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42047
I20260927 21:16:12.419222 19304 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:12.419400 19304 heartbeater.cc:511] Master 127.0.98.62:42047 requested a full tablet report, sending...
I20260927 21:16:12.419771 18985 ts_manager.cc:194] Registered new tserver with Master: 619561fcaed44c28aef3387f598df225 (127.0.98.2:43371)
I20260927 21:16:12.420168 18985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53407
W20260927 21:16:12.497095 19307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:12.497349 19307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:12.497380 19307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:12.498607 19307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:12.498660 19307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:12.498695 19307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:16:12.500113 19307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:12.500365 19307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:12.500667 19307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:12.505718 19315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:12.505708 19312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:12.505704 19313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:12.505959 19307 server_base.cc:1034] running on GCE node
I20260927 21:16:12.506173 19307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:12.506438 19307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:12.507597 19307 hybrid_clock.cc:648] HybridClock initialized: now 1790543772507581 us; error 30 us; skew 500 ppm
I20260927 21:16:12.508867 19307 webserver.cc:460] Webserver started at http://127.0.98.3:41785/ using document root <none> and password file <none>
I20260927 21:16:12.509107 19307 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:12.509162 19307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:12.509284 19307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:12.510425 19307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "1abd0592e3954af68365561c92fa3a76"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "34823255ca7f26c2c392c760b6f29323"
server_key_iv: "6d23d3260710468d78ace5341ed8abc6"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.510769 19307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1abd0592e3954af68365561c92fa3a76"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "34823255ca7f26c2c392c760b6f29323"
server_key_iv: "6d23d3260710468d78ace5341ed8abc6"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.512290 19307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:12.513190 19321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.513428 19307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:12.513518 19307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "1abd0592e3954af68365561c92fa3a76"
format_stamp: "Formatted at 2026-09-27 21:16:12 on dist-test-slave-r9x1"
server_key: "34823255ca7f26c2c392c760b6f29323"
server_key_iv: "6d23d3260710468d78ace5341ed8abc6"
server_key_version: "encryptionkey@0"
I20260927 21:16:12.513582 19307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:12.523216 19307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:12.523599 19307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:12.523757 19307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:12.524012 19307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:12.524360 19307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:12.524402 19307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.524433 19307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:12.524451 19307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:12.530429 19307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35845
I20260927 21:16:12.530470 19434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35845 every 8 connection(s)
I20260927 21:16:12.530803 19307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:16:12.536175 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.531535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36663 (local address 127.0.98.62:42047)
0927 21:16:12.531606 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.531608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.531832 (+ 224us) server_negotiation.cc:407] Connection header received
0927 21:16:12.532325 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.532326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.532337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.532385 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:12.533059 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.533645 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.534734 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.534934 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.534992 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.535859 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:12.535860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:12.535861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:12.535867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:12.535882 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:12.535887 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:12.535962 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.536071 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.536098 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:16:12.536296 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.531496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42047 (local address 127.0.98.3:36663)
0927 21:16:12.531707 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:12.531745 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:12.532230 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:12.532452 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:12.532456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:12.532464 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:12.532953 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.532956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.533755 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.533756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.534642 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.534645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.535007 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.535008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.535117 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.535696 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:16:12.535696 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:12.535726 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:16:12.535752 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:12.535972 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:12.535975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:12.535978 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:12.536030 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 21:16:12.536118 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260927 21:16:12.536890 19435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42047
I20260927 21:16:12.536988 19435 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:12.537170 19435 heartbeater.cc:511] Master 127.0.98.62:42047 requested a full tablet report, sending...
I20260927 21:16:12.537555 18985 ts_manager.cc:194] Registered new tserver with Master: 1abd0592e3954af68365561c92fa3a76 (127.0.98.3:35845)
I20260927 21:16:12.537928 18985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36663
I20260927 21:16:12.540431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 19307
I20260927 21:16:12.540545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:16:12.540730 392 external_mini_cluster.cc:1428] Setting key 1ea8187fe0550ce8e9b8ed4a9cd8b909
I20260927 21:16:12.541692 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:16:12.546114 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.543068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.0.98.62:42047)
0927 21:16:12.543111 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.543113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.543231 (+ 118us) server_negotiation.cc:407] Connection header received
0927 21:16:12.543267 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.543268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.543276 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.543322 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:12.543463 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.543942 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.544758 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.544968 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.545033 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.545786 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:12.545787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:12.545787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:12.545792 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:12.545806 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:12.545810 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:12.545903 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.546019 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.546030 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:16:12.547851 18985 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:16:12.549476 18985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:16:12.555905 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.552022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.0.98.3:35845)
0927 21:16:12.552295 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.552298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.552449 (+ 151us) server_negotiation.cc:407] Connection header received
0927 21:16:12.552495 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.552498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.552513 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.552582 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:12.552831 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.553611 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.554926 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.555472 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.555540 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.555639 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.555787 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.555824 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":71,"threads_started":1}
I20260927 21:16:12.555894 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.551914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35845 (local address 127.0.0.1:34416)
0927 21:16:12.552303 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:12.552331 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:12.552388 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:12.552743 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:12.552743 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:12.552745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:12.552774 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.552775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.553694 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.553695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.554850 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.554853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.555568 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.555569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.555664 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.555679 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:16:12.555747 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:16:12.555804 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":105,"threads_started":1}
I20260927 21:16:12.556318 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.552165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43371 (local address 127.0.0.1:42920)
0927 21:16:12.552345 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:12.552354 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:12.552397 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:12.552676 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:12.552677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:12.552679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:12.552724 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.552725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.554209 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.554210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.555180 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.555182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.556112 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.556113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.556176 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.556180 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:12.556205 (+ 25us) client_negotiation.cc:240] Negotiation successful
0927 21:16:12.556245 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":65,"threads_started":1}
I20260927 21:16:12.556650 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.552368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.0.98.2:43371)
0927 21:16:12.552542 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.552546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.552555 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:12.552588 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.552590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.552603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.552677 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:12.552788 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.554104 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.555256 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.556011 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.556422 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.556504 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.556532 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.556575 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260927 21:16:12.556981 19369 tablet_service.cc:1467] Processing CreateTablet for tablet 0026d6bac74f4e73bf6cfec049965290 (DEFAULT_TABLE table=test-table [id=91ff1aacc4a341f9bdb73a43ec2be221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:12.557394 19369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0026d6bac74f4e73bf6cfec049965290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:12.557323 19238 tablet_service.cc:1467] Processing CreateTablet for tablet 0026d6bac74f4e73bf6cfec049965290 (DEFAULT_TABLE table=test-table [id=91ff1aacc4a341f9bdb73a43ec2be221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:12.557668 19238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0026d6bac74f4e73bf6cfec049965290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:12.557855 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.551908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36471 (local address 127.0.0.1:52232)
0927 21:16:12.552122 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:12.552163 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:12.552232 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:12.552513 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:12.552516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:12.552518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:12.552590 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.552592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.555260 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.555261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.556296 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.556298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.557606 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.557607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.557675 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.557679 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:12.557739 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:16:12.557786 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
I20260927 21:16:12.557916 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.552019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.0.98.1:36471)
0927 21:16:12.552202 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.552207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.552219 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:16:12.552345 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.552347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.552365 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.552432 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:12.552672 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.554104 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.556550 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.557404 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.557484 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.557569 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.557782 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.557839 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260927 21:16:12.560068 19449 tablet_bootstrap.cc:492] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: Bootstrap starting.
I20260927 21:16:12.560770 19449 tablet_bootstrap.cc:654] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:12.560760 19450 tablet_bootstrap.cc:492] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: Bootstrap starting.
I20260927 21:16:12.560902 19107 tablet_service.cc:1467] Processing CreateTablet for tablet 0026d6bac74f4e73bf6cfec049965290 (DEFAULT_TABLE table=test-table [id=91ff1aacc4a341f9bdb73a43ec2be221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:12.561096 19449 log.cc:824] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:12.561226 19107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0026d6bac74f4e73bf6cfec049965290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:12.561399 19450 tablet_bootstrap.cc:654] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:12.561668 19450 log.cc:824] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:12.562414 19450 tablet_bootstrap.cc:492] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: No bootstrap required, opened a new log
I20260927 21:16:12.562489 19450 ts_tablet_manager.cc:1397] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:16:12.564530 19453 tablet_bootstrap.cc:492] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94: Bootstrap starting.
I20260927 21:16:12.564998 19450 raft_consensus.cc:348] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } }
I20260927 21:16:12.565135 19450 raft_consensus.cc:374] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:12.565135 19453 tablet_bootstrap.cc:654] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:12.565163 19450 raft_consensus.cc:729] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619561fcaed44c28aef3387f598df225, State: Initialized, Role: FOLLOWER
I20260927 21:16:12.565301 19450 consensus_queue.cc:260] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } }
I20260927 21:16:12.565573 19450 ts_tablet_manager.cc:1428] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 21:16:12.565621 19304 heartbeater.cc:503] Master 127.0.98.62:42047 was elected leader, sending a full tablet report...
I20260927 21:16:12.565474 19453 log.cc:824] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:12.566007 19449 tablet_bootstrap.cc:492] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: No bootstrap required, opened a new log
I20260927 21:16:12.566082 19449 ts_tablet_manager.cc:1397] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260927 21:16:12.566608 19453 tablet_bootstrap.cc:492] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94: No bootstrap required, opened a new log
I20260927 21:16:12.566674 19453 ts_tablet_manager.cc:1397] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:16:12.568408 19449 raft_consensus.cc:348] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } }
I20260927 21:16:12.568554 19449 raft_consensus.cc:374] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:12.568584 19449 raft_consensus.cc:729] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abd0592e3954af68365561c92fa3a76, State: Initialized, Role: FOLLOWER
I20260927 21:16:12.568701 19449 consensus_queue.cc:260] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } }
I20260927 21:16:12.568953 19449 ts_tablet_manager.cc:1428] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:16:12.568825 19453 raft_consensus.cc:348] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } }
I20260927 21:16:12.568996 19453 raft_consensus.cc:374] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:12.569048 19453 raft_consensus.cc:729] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4196e0939fd421c8f45589ed3ee7c94, State: Initialized, Role: FOLLOWER
I20260927 21:16:12.569073 19435 heartbeater.cc:503] Master 127.0.98.62:42047 was elected leader, sending a full tablet report...
I20260927 21:16:12.569164 19453 consensus_queue.cc:260] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } }
I20260927 21:16:12.569464 19453 ts_tablet_manager.cc:1428] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:16:12.569597 19173 heartbeater.cc:503] Master 127.0.98.62:42047 was elected leader, sending a full tablet report...
W20260927 21:16:12.663688 19305 tablet.cc:2369] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:16:12.768853 19174 tablet.cc:2369] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:16:12.781368 19436 tablet.cc:2369] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:16:12.802448 19457 raft_consensus.cc:482] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:16:12.802577 19457 raft_consensus.cc:504] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } }
I20260927 21:16:12.803010 19457 leader_election.cc:288] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1abd0592e3954af68365561c92fa3a76 (127.0.98.3:35845), 619561fcaed44c28aef3387f598df225 (127.0.98.2:43371)
I20260927 21:16:12.807252 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.803320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56741 (local address 127.0.98.3:35845)
0927 21:16:12.803406 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.803408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.803420 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:16:12.803574 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.803575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.803586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.803652 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:12.803861 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.804646 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.806236 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.806839 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.806916 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.806926 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.807133 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.807175 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:16:12.807252 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.803459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34925 (local address 127.0.98.2:43371)
0927 21:16:12.803504 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.803506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.803543 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:16:12.803628 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.803629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.803637 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.803689 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:12.803862 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.804660 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.806236 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.806847 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.806916 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.806926 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.807133 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.807175 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:16:12.807281 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.803390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43371 (local address 127.0.98.1:34925)
0927 21:16:12.803489 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:12.803498 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:12.803551 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:12.803697 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:12.803698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:12.803699 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:12.803754 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.803755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.804760 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.804761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.806129 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.806134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.806938 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.806940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.807041 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.807048 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:16:12.807115 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:16:12.807167 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":20,"thread_start_us":47,"threads_started":1}
I20260927 21:16:12.807288 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.803221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35845 (local address 127.0.98.1:56741)
0927 21:16:12.803345 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:12.803358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:12.803438 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:12.803668 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:12.803669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:12.803671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:12.803752 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.803755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.804757 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.804759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.806129 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:12.806134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.806939 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.806940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:12.807041 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.807050 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 21:16:12.807115 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 21:16:12.807167 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":39,"thread_start_us":65,"threads_started":1}
I20260927 21:16:12.807669 19389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026d6bac74f4e73bf6cfec049965290" candidate_uuid: "c4196e0939fd421c8f45589ed3ee7c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abd0592e3954af68365561c92fa3a76" is_pre_election: true
I20260927 21:16:12.807811 19389 raft_consensus.cc:2454] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4196e0939fd421c8f45589ed3ee7c94 in term 0.
I20260927 21:16:12.807943 19258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026d6bac74f4e73bf6cfec049965290" candidate_uuid: "c4196e0939fd421c8f45589ed3ee7c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619561fcaed44c28aef3387f598df225" is_pre_election: true
I20260927 21:16:12.808102 19258 raft_consensus.cc:2454] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4196e0939fd421c8f45589ed3ee7c94 in term 0.
I20260927 21:16:12.808085 19060 leader_election.cc:302] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abd0592e3954af68365561c92fa3a76, c4196e0939fd421c8f45589ed3ee7c94; no voters:
I20260927 21:16:12.808229 19457 raft_consensus.cc:2781] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:16:12.808290 19457 raft_consensus.cc:482] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:16:12.808331 19457 raft_consensus.cc:3037] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:12.809027 19457 raft_consensus.cc:504] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } }
I20260927 21:16:12.809150 19457 leader_election.cc:288] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [CANDIDATE]: Term 1 election: Requested vote from peers 1abd0592e3954af68365561c92fa3a76 (127.0.98.3:35845), 619561fcaed44c28aef3387f598df225 (127.0.98.2:43371)
I20260927 21:16:12.809378 19389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026d6bac74f4e73bf6cfec049965290" candidate_uuid: "c4196e0939fd421c8f45589ed3ee7c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abd0592e3954af68365561c92fa3a76"
I20260927 21:16:12.809413 19258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0026d6bac74f4e73bf6cfec049965290" candidate_uuid: "c4196e0939fd421c8f45589ed3ee7c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619561fcaed44c28aef3387f598df225"
I20260927 21:16:12.809484 19258 raft_consensus.cc:3037] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:12.809466 19389 raft_consensus.cc:3037] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:12.810248 19258 raft_consensus.cc:2454] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4196e0939fd421c8f45589ed3ee7c94 in term 1.
I20260927 21:16:12.810272 19389 raft_consensus.cc:2454] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4196e0939fd421c8f45589ed3ee7c94 in term 1.
I20260927 21:16:12.810477 19060 leader_election.cc:302] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abd0592e3954af68365561c92fa3a76, c4196e0939fd421c8f45589ed3ee7c94; no voters:
I20260927 21:16:12.810624 19457 raft_consensus.cc:2781] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:12.810792 19457 raft_consensus.cc:686] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 1 LEADER]: Becoming Leader. State: Replica: c4196e0939fd421c8f45589ed3ee7c94, State: Running, Role: LEADER
I20260927 21:16:12.810900 19457 consensus_queue.cc:237] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } }
I20260927 21:16:12.811951 18985 catalog_manager.cc:5495] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4196e0939fd421c8f45589ed3ee7c94 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c4196e0939fd421c8f45589ed3ee7c94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4196e0939fd421c8f45589ed3ee7c94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36471 } health_report { overall_health: HEALTHY } } }
I20260927 21:16:12.840898 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.838634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.0.98.1:36471)
0927 21:16:12.838698 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:12.838700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:12.838708 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:12.838746 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:12.838747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:12.838767 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:12.838827 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:12.838940 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.839411 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.840198 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:12.840403 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:12.840458 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:12.840566 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:12.840693 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:12.840765 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:12.840805 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 21:16:12.840815 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:16:12.842765 19258 raft_consensus.cc:1261] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [term 1 FOLLOWER]: Refusing update from remote peer c4196e0939fd421c8f45589ed3ee7c94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:12.842764 19389 raft_consensus.cc:1261] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [term 1 FOLLOWER]: Refusing update from remote peer c4196e0939fd421c8f45589ed3ee7c94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:12.843146 19460 consensus_queue.cc:1035] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abd0592e3954af68365561c92fa3a76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:12.843278 19457 consensus_queue.cc:1035] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619561fcaed44c28aef3387f598df225" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:12.846539 19465 mvcc.cc:205] Tried to move back new op lower bound from 7334067293561221120 to 7334067293434064896. Current Snapshot: MvccSnapshot[applied={T|T < 7334067293561221120 or (T in {7334067293561221120})}]
I20260927 21:16:12.846848 19464 mvcc.cc:205] Tried to move back new op lower bound from 7334067293561221120 to 7334067293434064896. Current Snapshot: MvccSnapshot[applied={T|T < 7334067293561221120}]
I20260927 21:16:12.847960 18985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:12.848047 18985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:12.848789 18985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e5001c8d6d240a3a2ee5f576a2072a3: Sending DeleteTablet for 3 replicas of tablet 0026d6bac74f4e73bf6cfec049965290
I20260927 21:16:12.849109 19369 tablet_service.cc:1514] Processing DeleteTablet for tablet 0026d6bac74f4e73bf6cfec049965290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:12 UTC) from {username='slave'} at 127.0.0.1:34416
I20260927 21:16:12.849123 19107 tablet_service.cc:1514] Processing DeleteTablet for tablet 0026d6bac74f4e73bf6cfec049965290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:12 UTC) from {username='slave'} at 127.0.0.1:52232
I20260927 21:16:12.849200 19238 tablet_service.cc:1514] Processing DeleteTablet for tablet 0026d6bac74f4e73bf6cfec049965290 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:12 UTC) from {username='slave'} at 127.0.0.1:42920
I20260927 21:16:12.849381 19474 tablet_replica.cc:330] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94: stopping tablet replica
I20260927 21:16:12.849479 19474 raft_consensus.cc:2229] T 0026d6bac74f4e73bf6cfec049965290 P c4196e0939fd421c8f45589ed3ee7c94 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:16:12.849638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 19045
I20260927 21:16:12.851356 19473 tablet_replica.cc:330] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: stopping tablet replica
I20260927 21:16:12.851490 19473 raft_consensus.cc:2229] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:16:12.851598 19473 raft_consensus.cc:2258] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:12.852077 19473 ts_tablet_manager.cc:1905] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:12.853434 19475 tablet_replica.cc:330] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: stopping tablet replica
I20260927 21:16:12.853556 19475 raft_consensus.cc:2229] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:16:12.853677 19475 raft_consensus.cc:2258] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:12.854041 19473 ts_tablet_manager.cc:1918] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:12.854091 19475 ts_tablet_manager.cc:1905] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:12.854118 19473 log.cc:1196] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/0026d6bac74f4e73bf6cfec049965290
I20260927 21:16:12.854339 19473 ts_tablet_manager.cc:1939] T 0026d6bac74f4e73bf6cfec049965290 P 1abd0592e3954af68365561c92fa3a76: Deleting consensus metadata
I20260927 21:16:12.854831 18969 catalog_manager.cc:4841] TS 1abd0592e3954af68365561c92fa3a76 (127.0.98.3:35845): tablet 0026d6bac74f4e73bf6cfec049965290 (table test-table [id=91ff1aacc4a341f9bdb73a43ec2be221]) successfully deleted
W20260927 21:16:12.855338 18971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36471 (error 108)
I20260927 21:16:12.855448 19475 ts_tablet_manager.cc:1918] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:12.855517 19475 log.cc:1196] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/0026d6bac74f4e73bf6cfec049965290
I20260927 21:16:12.855724 19475 ts_tablet_manager.cc:1939] T 0026d6bac74f4e73bf6cfec049965290 P 619561fcaed44c28aef3387f598df225: Deleting consensus metadata
I20260927 21:16:12.855762 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:12.855640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36471 (local address 127.0.0.1:52252)
0927 21:16:12.855692 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:12.855730 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:16:12.855854 18971 catalog_manager.cc:4584] TS c4196e0939fd421c8f45589ed3ee7c94 (127.0.98.1:36471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0026d6bac74f4e73bf6cfec049965290: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36471: connect: Connection refused (error 111)
I20260927 21:16:12.856132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 19176
I20260927 21:16:12.856130 18969 catalog_manager.cc:4841] TS 619561fcaed44c28aef3387f598df225 (127.0.98.2:43371): tablet 0026d6bac74f4e73bf6cfec049965290 (table test-table [id=91ff1aacc4a341f9bdb73a43ec2be221]) successfully deleted
I20260927 21:16:12.860558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 19307
I20260927 21:16:12.865172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 18954
2026-09-27T21:16:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (920 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:12 dist-test-slave-r9x1 krb5kdc[19479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:12 dist-test-slave-r9x1 krb5kdc[19479](info): set up 2 sockets
Sep 27 21:16:12 dist-test-slave-r9x1 krb5kdc[19479](info): commencing operation
krb5kdc: starting...
W20260927 21:16:13.828989 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:13 dist-test-slave-r9x1 krb5kdc[19479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:13.952715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39295
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:14.033862 19492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:14.034117 19492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:14.034153 19492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:14.035413 19492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:14.035465 19492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:14.035485 19492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:14.035501 19492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:14.035516 19492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:14.037115 19492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:14.037459 19492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:14.037688 19492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:14.042564 19500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:14.042564 19497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:14.042591 19498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:14.042645 19492 server_base.cc:1034] running on GCE node
I20260927 21:16:14.043061 19492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:14.043354 19492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:14.044530 19492 hybrid_clock.cc:648] HybridClock initialized: now 1790543774044510 us; error 43 us; skew 500 ppm
Sep 27 21:16:14 dist-test-slave-r9x1 krb5kdc[19479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:14.047626 19492 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:14.048246 19492 webserver.cc:460] Webserver started at http://127.0.98.62:34197/ using document root <none> and password file <none>
I20260927 21:16:14.048473 19492 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:14.048527 19492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:14.048652 19492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:14.050060 19492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "f627ec537f864ec48374f819e676e7db"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "36536990c398a8296505080980c1f148"
server_key_iv: "924b52f8d1a52eb785088d7213a188ba"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.050438 19492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "f627ec537f864ec48374f819e676e7db"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "36536990c398a8296505080980c1f148"
server_key_iv: "924b52f8d1a52eb785088d7213a188ba"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.051748 19492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:16:14.052676 19507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.052888 19492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:14.052979 19492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "f627ec537f864ec48374f819e676e7db"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "36536990c398a8296505080980c1f148"
server_key_iv: "924b52f8d1a52eb785088d7213a188ba"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.053049 19492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:14.060385 19492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:14.062767 19492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:14.062975 19492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:14.067124 19492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39295
I20260927 21:16:14.067207 19559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39295 every 8 connection(s)
I20260927 21:16:14.067550 19492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:14.068404 19560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:14.070809 19560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db: Bootstrap starting.
I20260927 21:16:14.071437 19560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:14.071702 19560 log.cc:824] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:14.072472 19560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db: No bootstrap required, opened a new log
I20260927 21:16:14.074223 19560 raft_consensus.cc:348] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f627ec537f864ec48374f819e676e7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39295 } }
I20260927 21:16:14.074379 19560 raft_consensus.cc:374] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:14.074401 19560 raft_consensus.cc:729] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f627ec537f864ec48374f819e676e7db, State: Initialized, Role: FOLLOWER
I20260927 21:16:14.074533 19560 consensus_queue.cc:260] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f627ec537f864ec48374f819e676e7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39295 } }
I20260927 21:16:14.074608 19560 raft_consensus.cc:388] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:14.074635 19560 raft_consensus.cc:482] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:14.074682 19560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:14.075266 19560 raft_consensus.cc:504] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f627ec537f864ec48374f819e676e7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39295 } }
I20260927 21:16:14.075362 19560 leader_election.cc:302] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f627ec537f864ec48374f819e676e7db; no voters:
I20260927 21:16:14.075517 19560 leader_election.cc:288] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:14.075604 19563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:14.075766 19563 raft_consensus.cc:686] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [term 1 LEADER]: Becoming Leader. State: Replica: f627ec537f864ec48374f819e676e7db, State: Running, Role: LEADER
I20260927 21:16:14.075776 19560 sys_catalog.cc:564] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:14.075872 19563 consensus_queue.cc:237] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f627ec537f864ec48374f819e676e7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39295 } }
I20260927 21:16:14.076277 19565 sys_catalog.cc:455] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [sys.catalog]: SysCatalogTable state changed. Reason: New leader f627ec537f864ec48374f819e676e7db. Latest consensus state: current_term: 1 leader_uuid: "f627ec537f864ec48374f819e676e7db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f627ec537f864ec48374f819e676e7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39295 } } }
I20260927 21:16:14.076381 19565 sys_catalog.cc:458] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:14.076413 19564 sys_catalog.cc:455] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f627ec537f864ec48374f819e676e7db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f627ec537f864ec48374f819e676e7db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39295 } } }
I20260927 21:16:14.076512 19564 sys_catalog.cc:458] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:14.077031 19572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:14.077564 19572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:14.077755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 19492
I20260927 21:16:14.077855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:14.078050 392 external_mini_cluster.cc:1428] Setting key 1c7943bae9b282034f2f2223aaebdb62
I20260927 21:16:14.079267 19572 catalog_manager.cc:1347] Generated new cluster ID: 946ee1fe4dad47b495e4306fbd8b2ecf
I20260927 21:16:14.079324 19572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:16:14 dist-test-slave-r9x1 krb5kdc[19479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:14.093312 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.078522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.0.98.62:39295)
0927 21:16:14.078681 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.078690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.078957 (+ 267us) server_negotiation.cc:407] Connection header received
0927 21:16:14.079061 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.079067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.079119 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.079245 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:14.080109 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.081626 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.082737 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.083458 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.083588 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.088095 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:14.088101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:14.088102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:14.088117 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:14.090911 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.091523 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.091531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.091532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.091925 (+ 393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.092211 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.092212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.092212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.092599 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:14.092741 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.092871 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.093025 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260927 21:16:14.108685 19572 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:14.109318 19572 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:14.120997 19572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db: Generated new TSK 0
I20260927 21:16:14.121223 19572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:16:14.152801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39295
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:14.237358 19586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:14.237574 19586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:14.237599 19586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:14.238835 19586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:14.238881 19586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:14.238909 19586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:16:14.240445 19586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:14.240677 19586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:14.240902 19586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:14.245961 19592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:14.245963 19591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:14.246166 19586 server_base.cc:1034] running on GCE node
W20260927 21:16:14.245985 19594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:14.246447 19586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:14.246668 19586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:14.247819 19586 hybrid_clock.cc:648] HybridClock initialized: now 1790543774247803 us; error 29 us; skew 500 ppm
Sep 27 21:16:14 dist-test-slave-r9x1 krb5kdc[19479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:14.250636 19586 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:16:14.251143 19586 webserver.cc:460] Webserver started at http://127.0.98.1:39547/ using document root <none> and password file <none>
I20260927 21:16:14.251353 19586 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:14.251401 19586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:14.251533 19586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:14.252652 19586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "dba3e9a0941d4f63812bb93aed8375de"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "35c60b73388ea30456821e97e822a422"
server_key_iv: "1c5fbbc58240db6048e3f1fd1889e033"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.253014 19586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dba3e9a0941d4f63812bb93aed8375de"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "35c60b73388ea30456821e97e822a422"
server_key_iv: "1c5fbbc58240db6048e3f1fd1889e033"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.254460 19586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:16:14.255302 19601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.255519 19586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:14.255611 19586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "dba3e9a0941d4f63812bb93aed8375de"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "35c60b73388ea30456821e97e822a422"
server_key_iv: "1c5fbbc58240db6048e3f1fd1889e033"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.255693 19586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:14.263070 19586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:14.265182 19586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:14.265359 19586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:14.265586 19586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:14.265883 19586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:14.265918 19586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.265947 19586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:14.265971 19586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.272082 19586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42907
I20260927 21:16:14.272152 19714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42907 every 8 connection(s)
I20260927 21:16:14.272475 19586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:16:14.278085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 19586
I20260927 21:16:14.278206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:16:14.278431 392 external_mini_cluster.cc:1428] Setting key 1fec215912a4892e7ca834bdc2088e08
Sep 27 21:16:14 dist-test-slave-r9x1 krb5kdc[19479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:14.282226 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.273248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52525 (local address 127.0.98.62:39295)
0927 21:16:14.273348 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.273350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.273491 (+ 141us) server_negotiation.cc:407] Connection header received
0927 21:16:14.274409 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.274410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.274425 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.274475 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:14.275081 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.275597 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.276707 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.276926 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.276981 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.279548 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:14.279550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:14.279550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:14.279558 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:14.280833 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.281246 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.281247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.281248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.281334 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.281576 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.281576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.281577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.281692 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:14.281748 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.282039 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.282136 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:16:14.282416 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.273168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39295 (local address 127.0.98.1:52525)
0927 21:16:14.273348 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.273386 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:14.274310 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:14.274540 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:14.274543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:14.274813 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:14.275013 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.275018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.275682 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.275683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.276605 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.276608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.277012 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.277013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.277122 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.277672 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:14.277681 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:14.279449 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:14.280922 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:14.280931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:14.280934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:14.281170 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:14.281392 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:14.281393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:14.281393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:14.281515 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:14.281753 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:14.281761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:14.281907 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 21:16:14.281999 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:16:14.282185 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
I20260927 21:16:14.282963 19715 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39295
I20260927 21:16:14.283066 19715 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:14.283254 19715 heartbeater.cc:511] Master 127.0.98.62:39295 requested a full tablet report, sending...
I20260927 21:16:14.283809 19524 ts_manager.cc:194] Registered new tserver with Master: dba3e9a0941d4f63812bb93aed8375de (127.0.98.1:42907)
I20260927 21:16:14.284444 19524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52525
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:16:14.305534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39295
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:14.387689 19720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:14.387920 19720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:14.387949 19720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:14.389199 19720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:14.389243 19720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:14.389302 19720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:16:14.391001 19720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:14.391237 19720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:14.391448 19720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:14.396608 19726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:14.396596 19728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:14.396600 19725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:14.396617 19720 server_base.cc:1034] running on GCE node
I20260927 21:16:14.397043 19720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:14.397346 19720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:14.398618 19720 hybrid_clock.cc:648] HybridClock initialized: now 1790543774398604 us; error 30 us; skew 500 ppm
Sep 27 21:16:14 dist-test-slave-r9x1 krb5kdc[19479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:14.401659 19720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:16:14.402187 19720 webserver.cc:460] Webserver started at http://127.0.98.2:37645/ using document root <none> and password file <none>
I20260927 21:16:14.402431 19720 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:14.402484 19720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:14.402618 19720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:14.403736 19720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "741d108f72d74241b49ed8075ec1a7f4"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "7453f12f5907221ec10666ef5c87204b"
server_key_iv: "b686ef1b0261b50c0e6a19d379307d31"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.404107 19720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "741d108f72d74241b49ed8075ec1a7f4"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "7453f12f5907221ec10666ef5c87204b"
server_key_iv: "b686ef1b0261b50c0e6a19d379307d31"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.405609 19720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:16:14.406504 19735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.406710 19720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:14.406803 19720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "741d108f72d74241b49ed8075ec1a7f4"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "7453f12f5907221ec10666ef5c87204b"
server_key_iv: "b686ef1b0261b50c0e6a19d379307d31"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.406886 19720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:14.429858 19720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:14.432204 19720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:14.432351 19720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:14.432591 19720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:14.432904 19720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:14.432945 19720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.432976 19720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:14.432993 19720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.438577 19720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33215
I20260927 21:16:14.438654 19848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33215 every 8 connection(s)
I20260927 21:16:14.438970 19720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:16:14.440793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 19720
I20260927 21:16:14.440915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:16:14.441102 392 external_mini_cluster.cc:1428] Setting key 5e79db05732d0834eb2c4cc576ad0a61
Sep 27 21:16:14 dist-test-slave-r9x1 krb5kdc[19479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:16:14.449977 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.439698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57375 (local address 127.0.98.62:39295)
0927 21:16:14.439775 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.439777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.439983 (+ 206us) server_negotiation.cc:407] Connection header received
0927 21:16:14.440964 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.440965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.440979 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.441033 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:14.441810 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.442503 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.443862 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.444060 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.444129 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.446690 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:14.446692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:14.446692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:14.446700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:14.448245 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.448829 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.448830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.448831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.448895 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.449199 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.449199 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.449200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.449388 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:14.449445 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.449816 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.449907 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:16:14.450155 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.439665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39295 (local address 127.0.98.2:57375)
0927 21:16:14.439844 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.439884 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:14.440838 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:14.441109 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:14.441113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:14.441490 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:14.441701 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.441707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.442607 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.442608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.443756 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.443759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.444164 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.444165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.444303 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.444843 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:14.444853 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:14.446579 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:14.448352 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:14.448365 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:14.448368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:14.448686 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:14.448969 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:14.448970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:14.448971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:14.449126 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:14.449453 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:14.449461 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:14.449656 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 21:16:14.449783 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 21:16:14.449950 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260927 21:16:14.450673 19849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39295
I20260927 21:16:14.450779 19849 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:14.451000 19849 heartbeater.cc:511] Master 127.0.98.62:39295 requested a full tablet report, sending...
I20260927 21:16:14.451452 19524 ts_manager.cc:194] Registered new tserver with Master: 741d108f72d74241b49ed8075ec1a7f4 (127.0.98.2:33215)
I20260927 21:16:14.451822 19524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:16:14.469252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39295
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:14.555948 19854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:14.556185 19854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:14.556219 19854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:14.557564 19854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:14.557615 19854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:14.557655 19854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:16:14.559604 19854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:14.559890 19854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:14.560127 19854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:14.565692 19860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:14.565701 19862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:14.565675 19859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:14.565871 19854 server_base.cc:1034] running on GCE node
I20260927 21:16:14.566097 19854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:14.566339 19854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:14.567482 19854 hybrid_clock.cc:648] HybridClock initialized: now 1790543774567515 us; error 100 us; skew 500 ppm
Sep 27 21:16:14 dist-test-slave-r9x1 krb5kdc[19479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:14.570106 19854 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:16:14.570560 19854 webserver.cc:460] Webserver started at http://127.0.98.3:39047/ using document root <none> and password file <none>
I20260927 21:16:14.570777 19854 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:14.570825 19854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:14.570961 19854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:14.572024 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "bb7b9120a8394a978f3706315eba7a55"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "b3f0fdd3e5376231f5371811af310041"
server_key_iv: "6a85fca573acd85b372e7255589a2181"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.572376 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bb7b9120a8394a978f3706315eba7a55"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "b3f0fdd3e5376231f5371811af310041"
server_key_iv: "6a85fca573acd85b372e7255589a2181"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.574016 19854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:14.575050 19869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.575341 19854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:14.575441 19854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "bb7b9120a8394a978f3706315eba7a55"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "b3f0fdd3e5376231f5371811af310041"
server_key_iv: "6a85fca573acd85b372e7255589a2181"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.575515 19854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:14.591578 19854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:14.593930 19854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:14.594076 19854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:14.594288 19854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:14.594590 19854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:14.594624 19854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.594655 19854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:14.594679 19854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.600363 19854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34639
I20260927 21:16:14.600419 19982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34639 every 8 connection(s)
I20260927 21:16:14.600792 19854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:16:14 dist-test-slave-r9x1 krb5kdc[19479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:14.609865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 19854
I20260927 21:16:14.609985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:16:14.610189 392 external_mini_cluster.cc:1428] Setting key 99dad7f9cf1d481bdf1d323b851b2a6b
I20260927 21:16:14.611732 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.601557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35861 (local address 127.0.98.62:39295)
0927 21:16:14.601631 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.601633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.601903 (+ 270us) server_negotiation.cc:407] Connection header received
0927 21:16:14.602877 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.602879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.602891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.602957 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:14.603700 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.604256 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.605410 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.605630 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.605701 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.608292 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:14.608293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:14.608294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:14.608302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:14.610228 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.610817 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.610819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.610819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.610888 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.611144 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.611145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.611145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.611280 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:14.611334 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.611596 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.611664 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:16:14.611956 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.601530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39295 (local address 127.0.98.3:35861)
0927 21:16:14.601695 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.601740 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:14.602703 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:14.603067 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:14.603071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:14.603360 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:14.603588 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.603592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.604361 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.604363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.605306 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.605310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.605758 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.605760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.605893 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.606596 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:14.606606 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:14.608159 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:14.610352 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:14.610364 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:14.610367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:14.610674 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:14.610939 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:14.610940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:14.610940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:14.611089 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:14.611346 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:14.611351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:14.611485 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 21:16:14.611573 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:16:14.611714 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260927 21:16:14.612535 19983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39295
I20260927 21:16:14.612645 19983 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:14.612833 19983 heartbeater.cc:511] Master 127.0.98.62:39295 requested a full tablet report, sending...
I20260927 21:16:14.613219 19524 ts_manager.cc:194] Registered new tserver with Master: bb7b9120a8394a978f3706315eba7a55 (127.0.98.3:34639)
I20260927 21:16:14.613606 19524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35861
I20260927 21:16:14.622231 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:16:14.630215 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.623565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.0.98.62:39295)
0927 21:16:14.623650 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.623652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.623722 (+ 70us) server_negotiation.cc:407] Connection header received
0927 21:16:14.623769 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.623770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.623779 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.623844 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:14.624377 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.624881 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.625641 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.625833 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.625887 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.627278 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:14.627280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:14.627280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:14.627287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:14.628666 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.629229 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.629230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.629230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.629326 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:14.629623 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:14.629624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:14.629624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:14.629805 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:14.629897 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.630050 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.630125 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:16:14.631944 19524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:16:14.633625 19524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:16:14.640264 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.636292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34639 (local address 127.0.0.1:59914)
0927 21:16:14.636672 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.636680 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:14.636723 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:14.636986 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:14.636987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:14.636989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:14.637032 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.637033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.638224 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.638225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.639006 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.639008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.639931 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.639932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.640025 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.640041 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 21:16:14.640110 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:16:14.640173 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":62,"threads_started":1}
I20260927 21:16:14.640439 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.636189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.0.98.2:33215)
0927 21:16:14.636517 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.636522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.636530 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:14.636568 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.636570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.636584 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.636651 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:14.636842 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.637774 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.639233 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.639934 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.639991 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.640041 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.640308 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.640357 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":68,"threads_started":1}
I20260927 21:16:14.640441 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.636482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.0.98.3:34639)
0927 21:16:14.636828 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.636833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.636841 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:14.636875 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.636877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.636891 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.636964 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:14.637093 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.638126 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.639110 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.639827 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.639908 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.639975 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.640316 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.640362 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":97,"threads_started":1}
I20260927 21:16:14.640726 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.636082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33215 (local address 127.0.0.1:34008)
0927 21:16:14.636332 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.636340 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:14.636378 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:14.636739 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:14.636740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:14.636742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:14.636775 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.636776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.637863 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.637864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.639126 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.639129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.640076 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.640077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.640144 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.640148 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:14.640195 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:16:14.640241 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260927 21:16:14.640730 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.636071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42907 (local address 127.0.0.1:41356)
0927 21:16:14.636261 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.636304 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:14.636373 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:14.636570 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:14.636573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:14.636575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:14.636656 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.636659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.638126 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.638127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.639179 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.639181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.640135 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.640135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.640186 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.640189 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:14.640236 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:16:14.640275 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260927 21:16:14.641415 19917 tablet_service.cc:1467] Processing CreateTablet for tablet 4e56c61b91164b939044270c099c343b (DEFAULT_TABLE table=test-table [id=42735c9bded8417c948045470aff197b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:14.641553 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.636201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.0.98.1:42907)
0927 21:16:14.636393 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.636398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.636413 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:16:14.636458 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.636460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.636475 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.636565 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:14.636740 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.638020 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.639267 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.640043 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.640119 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.640175 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.640250 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.640291 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":75,"threads_started":1}
I20260927 21:16:14.641786 19917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e56c61b91164b939044270c099c343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:14.641957 19783 tablet_service.cc:1467] Processing CreateTablet for tablet 4e56c61b91164b939044270c099c343b (DEFAULT_TABLE table=test-table [id=42735c9bded8417c948045470aff197b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:14.642122 19649 tablet_service.cc:1467] Processing CreateTablet for tablet 4e56c61b91164b939044270c099c343b (DEFAULT_TABLE table=test-table [id=42735c9bded8417c948045470aff197b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:14.642274 19783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e56c61b91164b939044270c099c343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:14.642446 19649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e56c61b91164b939044270c099c343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:14.644929 20000 tablet_bootstrap.cc:492] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de: Bootstrap starting.
I20260927 21:16:14.645318 20001 tablet_bootstrap.cc:492] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: Bootstrap starting.
I20260927 21:16:14.645504 20002 tablet_bootstrap.cc:492] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4: Bootstrap starting.
I20260927 21:16:14.645947 20001 tablet_bootstrap.cc:654] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:14.645987 20000 tablet_bootstrap.cc:654] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:14.646085 20002 tablet_bootstrap.cc:654] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:14.646454 20001 log.cc:824] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:14.646541 20000 log.cc:824] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:14.646600 20002 log.cc:824] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:14.647326 20000 tablet_bootstrap.cc:492] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de: No bootstrap required, opened a new log
I20260927 21:16:14.647326 20002 tablet_bootstrap.cc:492] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4: No bootstrap required, opened a new log
I20260927 21:16:14.647404 20000 ts_tablet_manager.cc:1397] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 21:16:14.647413 20002 ts_tablet_manager.cc:1397] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:16:14.649483 20002 raft_consensus.cc:348] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } }
I20260927 21:16:14.649483 20000 raft_consensus.cc:348] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } }
I20260927 21:16:14.649689 20000 raft_consensus.cc:374] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:14.649689 20002 raft_consensus.cc:374] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:14.649737 20000 raft_consensus.cc:729] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba3e9a0941d4f63812bb93aed8375de, State: Initialized, Role: FOLLOWER
I20260927 21:16:14.649737 20002 raft_consensus.cc:729] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741d108f72d74241b49ed8075ec1a7f4, State: Initialized, Role: FOLLOWER
I20260927 21:16:14.649863 20002 consensus_queue.cc:260] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } }
I20260927 21:16:14.649863 20000 consensus_queue.cc:260] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } }
I20260927 21:16:14.649977 20001 tablet_bootstrap.cc:492] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: No bootstrap required, opened a new log
I20260927 21:16:14.650081 20001 ts_tablet_manager.cc:1397] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260927 21:16:14.650162 20002 ts_tablet_manager.cc:1428] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:16:14.650162 20000 ts_tablet_manager.cc:1428] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 21:16:14.650200 19849 heartbeater.cc:503] Master 127.0.98.62:39295 was elected leader, sending a full tablet report...
I20260927 21:16:14.650280 19715 heartbeater.cc:503] Master 127.0.98.62:39295 was elected leader, sending a full tablet report...
I20260927 21:16:14.652073 20001 raft_consensus.cc:348] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } }
I20260927 21:16:14.652231 20001 raft_consensus.cc:374] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:14.652253 20001 raft_consensus.cc:729] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7b9120a8394a978f3706315eba7a55, State: Initialized, Role: FOLLOWER
I20260927 21:16:14.652359 20001 consensus_queue.cc:260] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } }
I20260927 21:16:14.652637 20001 ts_tablet_manager.cc:1428] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:16:14.652712 19983 heartbeater.cc:503] Master 127.0.98.62:39295 was elected leader, sending a full tablet report...
W20260927 21:16:14.689543 19850 tablet.cc:2369] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:16:14.778577 19716 tablet.cc:2369] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:16:14.785511 20007 raft_consensus.cc:482] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:16:14.785640 20007 raft_consensus.cc:504] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } }
I20260927 21:16:14.785964 20007 leader_election.cc:288] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 741d108f72d74241b49ed8075ec1a7f4 (127.0.98.2:33215), bb7b9120a8394a978f3706315eba7a55 (127.0.98.3:34639)
I20260927 21:16:14.789381 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.786307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40963 (local address 127.0.98.2:33215)
0927 21:16:14.786347 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.786349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.786465 (+ 116us) server_negotiation.cc:407] Connection header received
0927 21:16:14.786540 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.786541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.786552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.786605 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:14.786789 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.787567 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.788644 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.789029 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.789091 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.789103 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.789250 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.789312 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:16:14.789418 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.786228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50679 (local address 127.0.98.3:34639)
0927 21:16:14.786281 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.786283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.786395 (+ 112us) server_negotiation.cc:407] Connection header received
0927 21:16:14.786498 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.786499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.786509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.786589 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:14.786790 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.787540 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.788657 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.789060 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.789125 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.789136 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.789306 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.789350 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:16:14.789427 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.786182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33215 (local address 127.0.98.1:40963)
0927 21:16:14.786418 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.786426 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:14.786474 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:14.786615 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:14.786615 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:14.786616 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:14.786726 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.786728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.787652 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.787654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.788549 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.788553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.789114 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.789115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.789185 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.789188 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:14.789233 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:16:14.789303 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":27,"thread_start_us":73,"threads_started":1}
I20260927 21:16:14.789444 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.786168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34639 (local address 127.0.98.1:50679)
0927 21:16:14.786334 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.786345 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:14.786414 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:14.786607 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:14.786609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:14.786611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:14.786714 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.786717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.787709 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.787711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.788559 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:14.788562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.789146 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.789147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:14.789218 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.789222 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:14.789302 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:16:14.789343 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260927 21:16:14.789901 19803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e56c61b91164b939044270c099c343b" candidate_uuid: "dba3e9a0941d4f63812bb93aed8375de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741d108f72d74241b49ed8075ec1a7f4" is_pre_election: true
I20260927 21:16:14.789954 19937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e56c61b91164b939044270c099c343b" candidate_uuid: "dba3e9a0941d4f63812bb93aed8375de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7b9120a8394a978f3706315eba7a55" is_pre_election: true
I20260927 21:16:14.790082 19937 raft_consensus.cc:2454] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba3e9a0941d4f63812bb93aed8375de in term 0.
I20260927 21:16:14.790082 19803 raft_consensus.cc:2454] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba3e9a0941d4f63812bb93aed8375de in term 0.
I20260927 21:16:14.790310 19603 leader_election.cc:302] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741d108f72d74241b49ed8075ec1a7f4, dba3e9a0941d4f63812bb93aed8375de; no voters:
I20260927 21:16:14.790464 20007 raft_consensus.cc:2781] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:16:14.790526 20007 raft_consensus.cc:482] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:16:14.790544 20007 raft_consensus.cc:3037] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:14.791188 20007 raft_consensus.cc:504] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } }
I20260927 21:16:14.791302 20007 leader_election.cc:288] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [CANDIDATE]: Term 1 election: Requested vote from peers 741d108f72d74241b49ed8075ec1a7f4 (127.0.98.2:33215), bb7b9120a8394a978f3706315eba7a55 (127.0.98.3:34639)
I20260927 21:16:14.791472 19803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e56c61b91164b939044270c099c343b" candidate_uuid: "dba3e9a0941d4f63812bb93aed8375de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741d108f72d74241b49ed8075ec1a7f4"
I20260927 21:16:14.791509 19937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e56c61b91164b939044270c099c343b" candidate_uuid: "dba3e9a0941d4f63812bb93aed8375de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7b9120a8394a978f3706315eba7a55"
I20260927 21:16:14.791554 19803 raft_consensus.cc:3037] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:14.791579 19937 raft_consensus.cc:3037] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:14.792253 19937 raft_consensus.cc:2454] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba3e9a0941d4f63812bb93aed8375de in term 1.
I20260927 21:16:14.792264 19803 raft_consensus.cc:2454] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba3e9a0941d4f63812bb93aed8375de in term 1.
I20260927 21:16:14.792434 19605 leader_election.cc:302] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb7b9120a8394a978f3706315eba7a55, dba3e9a0941d4f63812bb93aed8375de; no voters:
I20260927 21:16:14.792557 20007 raft_consensus.cc:2781] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:14.792691 20007 raft_consensus.cc:686] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [term 1 LEADER]: Becoming Leader. State: Replica: dba3e9a0941d4f63812bb93aed8375de, State: Running, Role: LEADER
I20260927 21:16:14.792778 20007 consensus_queue.cc:237] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } }
I20260927 21:16:14.793704 19524 catalog_manager.cc:5495] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de reported cstate change: term changed from 0 to 1, leader changed from <none> to dba3e9a0941d4f63812bb93aed8375de (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dba3e9a0941d4f63812bb93aed8375de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba3e9a0941d4f63812bb93aed8375de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 } health_report { overall_health: UNKNOWN } } }
I20260927 21:16:14.828625 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.826275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.0.98.1:42907)
0927 21:16:14.826333 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:14.826335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:14.826343 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:14.826379 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:14.826380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:14.826400 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:14.826457 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:14.826598 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.827110 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.827891 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:14.828087 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:14.828150 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:14.828335 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:14.828445 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:14.828516 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:14.828532 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 21:16:14.828554 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:16:14.830673 19937 raft_consensus.cc:1261] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [term 1 FOLLOWER]: Refusing update from remote peer dba3e9a0941d4f63812bb93aed8375de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:14.830706 19803 raft_consensus.cc:1261] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4 [term 1 FOLLOWER]: Refusing update from remote peer dba3e9a0941d4f63812bb93aed8375de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:14.831003 20007 consensus_queue.cc:1035] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7b9120a8394a978f3706315eba7a55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:14.831162 20011 consensus_queue.cc:1035] T 4e56c61b91164b939044270c099c343b P dba3e9a0941d4f63812bb93aed8375de [LEADER]: Connected to new peer: Peer: permanent_uuid: "741d108f72d74241b49ed8075ec1a7f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:14.833418 20014 mvcc.cc:205] Tried to move back new op lower bound from 7334067301703221248 to 7334067301552037888. Current Snapshot: MvccSnapshot[applied={T|T < 7334067301703221248}]
I20260927 21:16:14.835966 19523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:14.836067 19523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:14.836882 19523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f627ec537f864ec48374f819e676e7db: Sending DeleteTablet for 3 replicas of tablet 4e56c61b91164b939044270c099c343b
I20260927 21:16:14.837258 19783 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e56c61b91164b939044270c099c343b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34008
I20260927 21:16:14.837291 19649 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e56c61b91164b939044270c099c343b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41356
I20260927 21:16:14.837345 19917 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e56c61b91164b939044270c099c343b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59914
I20260927 21:16:14.837646 20026 tablet_replica.cc:330] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: stopping tablet replica
I20260927 21:16:14.837761 20026 raft_consensus.cc:2229] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:16:14.837791 20024 tablet_replica.cc:330] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4: stopping tablet replica
I20260927 21:16:14.837889 20026 raft_consensus.cc:2258] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:14.837896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 19586
I20260927 21:16:14.840153 20026 ts_tablet_manager.cc:1905] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:14.841849 20026 ts_tablet_manager.cc:1918] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:14.841919 20026 log.cc:1196] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/4e56c61b91164b939044270c099c343b
I20260927 21:16:14.842155 20026 ts_tablet_manager.cc:1939] T 4e56c61b91164b939044270c099c343b P bb7b9120a8394a978f3706315eba7a55: Deleting consensus metadata
I20260927 21:16:14.842599 19511 catalog_manager.cc:4841] TS bb7b9120a8394a978f3706315eba7a55 (127.0.98.3:34639): tablet 4e56c61b91164b939044270c099c343b (table test-table [id=42735c9bded8417c948045470aff197b]) successfully deleted
W20260927 21:16:14.843250 19511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42907 (error 108)
I20260927 21:16:14.843782 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.843652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42907 (local address 127.0.0.1:41364)
0927 21:16:14.843707 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.843746 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:16:14.843820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 19720
I20260927 21:16:14.843811 20024 raft_consensus.cc:2229] T 4e56c61b91164b939044270c099c343b P 741d108f72d74241b49ed8075ec1a7f4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 21:16:14.843909 19511 catalog_manager.cc:4584] TS dba3e9a0941d4f63812bb93aed8375de (127.0.98.1:42907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e56c61b91164b939044270c099c343b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42907: connect: Connection refused (error 111)
I20260927 21:16:14.848250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 19854
I20260927 21:16:14.848349 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:14.848227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33215 (local address 127.0.0.1:34022)
0927 21:16:14.848286 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:14.848312 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:16:14.852818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 19492
2026-09-27T21:16:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1988 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T21:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:14Z Disabled control of system clock
I20260927 21:16:14.879390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36321
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:16:14.962805 20031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:14.963048 20031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:14.963071 20031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:14.964486 20031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:14.964560 20031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:14.964574 20031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:14.964586 20031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:14.964597 20031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:14.966173 20031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36321
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:14.966405 20031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:14.966665 20031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:14.971418 20037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:14.971449 20036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:14.971369 20039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:14.971625 20031 server_base.cc:1034] running on GCE node
I20260927 21:16:14.971882 20031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:14.972216 20031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:14.973342 20031 hybrid_clock.cc:648] HybridClock initialized: now 1790543774973320 us; error 35 us; skew 500 ppm
I20260927 21:16:14.974583 20031 webserver.cc:460] Webserver started at http://127.0.98.62:36245/ using document root <none> and password file <none>
I20260927 21:16:14.974824 20031 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:14.974872 20031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:14.975039 20031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:14.976133 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "f4d504f5599a4994be2d44c37dcc54bf"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "54e65b015ff443e8e50aaf5b41ec0150"
server_key_iv: "d7c5a727121947fb92e854faf6649006"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.976469 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4d504f5599a4994be2d44c37dcc54bf"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "54e65b015ff443e8e50aaf5b41ec0150"
server_key_iv: "d7c5a727121947fb92e854faf6649006"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.977897 20031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:14.978677 20045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:14.978830 20031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:16:14.978951 20031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "f4d504f5599a4994be2d44c37dcc54bf"
format_stamp: "Formatted at 2026-09-27 21:16:14 on dist-test-slave-r9x1"
server_key: "54e65b015ff443e8e50aaf5b41ec0150"
server_key_iv: "d7c5a727121947fb92e854faf6649006"
server_key_version: "encryptionkey@0"
I20260927 21:16:14.979024 20031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:15.001457 20031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:15.001792 20031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:15.001930 20031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:15.005682 20031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36321
I20260927 21:16:15.005726 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36321 every 8 connection(s)
I20260927 21:16:15.006062 20031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:15.007021 20098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:15.009470 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf: Bootstrap starting.
I20260927 21:16:15.010066 20098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:15.010351 20098 log.cc:824] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:15.011188 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf: No bootstrap required, opened a new log
I20260927 21:16:15.013089 20098 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d504f5599a4994be2d44c37dcc54bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } }
I20260927 21:16:15.013298 20098 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:15.013337 20098 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d504f5599a4994be2d44c37dcc54bf, State: Initialized, Role: FOLLOWER
I20260927 21:16:15.013481 20098 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d504f5599a4994be2d44c37dcc54bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } }
I20260927 21:16:15.013561 20098 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:15.013584 20098 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:15.013616 20098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:15.014127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 20031
I20260927 21:16:15.014238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:15.014298 20098 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d504f5599a4994be2d44c37dcc54bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } }
I20260927 21:16:15.014417 20098 leader_election.cc:302] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4d504f5599a4994be2d44c37dcc54bf; no voters:
I20260927 21:16:15.014432 392 external_mini_cluster.cc:1428] Setting key 7ecc712b75de69c2cf2085716bc62b7a
I20260927 21:16:15.014602 20098 leader_election.cc:288] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:15.014659 20101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:15.014830 20101 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [term 1 LEADER]: Becoming Leader. State: Replica: f4d504f5599a4994be2d44c37dcc54bf, State: Running, Role: LEADER
I20260927 21:16:15.014886 20098 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:15.015327 20101 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d504f5599a4994be2d44c37dcc54bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } }
I20260927 21:16:15.016338 20102 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4d504f5599a4994be2d44c37dcc54bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d504f5599a4994be2d44c37dcc54bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } } }
I20260927 21:16:15.016799 20102 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:15.016695 20103 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4d504f5599a4994be2d44c37dcc54bf. Latest consensus state: current_term: 1 leader_uuid: "f4d504f5599a4994be2d44c37dcc54bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4d504f5599a4994be2d44c37dcc54bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } } }
I20260927 21:16:15.017045 20103 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:15.017241 20115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:15.017760 20115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:15.019631 20115 catalog_manager.cc:1347] Generated new cluster ID: 786d5b755ed3418f989a7a9262de6f66
I20260927 21:16:15.019685 20115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:15.022032 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.015179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.0.98.62:36321)
0927 21:16:15.015356 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.015366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.015402 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:16:15.015553 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.015563 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.015647 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.015800 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:15.016061 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.018315 (+ 2254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.019450 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.020046 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.020165 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.021123 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:15.021128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:15.021128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:15.021369 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:15.021418 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:15.021435 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:15.021495 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.021716 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.021806 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260927 21:16:15.048048 20115 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:15.048575 20115 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:15.058511 20115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf: Generated new TSK 0
I20260927 21:16:15.058712 20115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:15.062944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36321
--builtin_ntp_servers=127.0.98.20:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:16:15.144972 20122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:15.145197 20122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:15.145219 20122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:15.146438 20122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:15.146483 20122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:15.146508 20122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:16:15.147951 20122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:15.148173 20122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:15.148444 20122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:15.153509 20127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:15.153503 20130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:15.153502 20128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:15.153719 20122 server_base.cc:1034] running on GCE node
I20260927 21:16:15.153920 20122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:15.154171 20122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:15.155349 20122 hybrid_clock.cc:648] HybridClock initialized: now 1790543775155329 us; error 37 us; skew 500 ppm
I20260927 21:16:15.156718 20122 webserver.cc:460] Webserver started at http://127.0.98.1:44673/ using document root <none> and password file <none>
I20260927 21:16:15.156998 20122 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:15.157052 20122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:15.157178 20122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:15.158345 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/data/instance:
uuid: "a39ad32b567248779e0c46406b0794b1"
format_stamp: "Formatted at 2026-09-27 21:16:15 on dist-test-slave-r9x1"
server_key: "3a6fee163ba3e74cc17fc100dc5210e1"
server_key_iv: "ade27c0a0c76597e0c8a39b263ab1a5b"
server_key_version: "encryptionkey@0"
I20260927 21:16:15.158710 20122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a39ad32b567248779e0c46406b0794b1"
format_stamp: "Formatted at 2026-09-27 21:16:15 on dist-test-slave-r9x1"
server_key: "3a6fee163ba3e74cc17fc100dc5210e1"
server_key_iv: "ade27c0a0c76597e0c8a39b263ab1a5b"
server_key_version: "encryptionkey@0"
I20260927 21:16:15.160054 20122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:16:15.160821 20136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:15.161011 20122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:16:15.161096 20122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/wal
uuid: "a39ad32b567248779e0c46406b0794b1"
format_stamp: "Formatted at 2026-09-27 21:16:15 on dist-test-slave-r9x1"
server_key: "3a6fee163ba3e74cc17fc100dc5210e1"
server_key_iv: "ade27c0a0c76597e0c8a39b263ab1a5b"
server_key_version: "encryptionkey@0"
I20260927 21:16:15.161165 20122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:15.180845 20122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:15.181176 20122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:15.181353 20122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:15.181597 20122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:15.181994 20122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:15.182029 20122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:15.182058 20122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:15.182075 20122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:15.187723 20122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43615
I20260927 21:16:15.187776 20249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43615 every 8 connection(s)
I20260927 21:16:15.188139 20122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:16:15.193671 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.188878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50701 (local address 127.0.98.62:36321)
0927 21:16:15.188973 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.188975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.189154 (+ 179us) server_negotiation.cc:407] Connection header received
0927 21:16:15.189722 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.189724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.189737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.189792 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:15.190520 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.191044 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.192188 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.192408 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.192471 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.193433 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:15.193434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:15.193435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:15.193439 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:15.193455 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:15.193459 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:15.193515 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.193573 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.193604 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:16:15.193876 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.188846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36321 (local address 127.0.98.1:50701)
0927 21:16:15.189036 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:15.189071 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:15.189620 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:15.189849 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:15.189853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:15.189861 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:15.190399 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.190411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.191200 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.191201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.192032 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.192035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.192530 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.192531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.192662 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.193250 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:16:15.193250 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:15.193324 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:16:15.193353 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:15.193516 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:15.193518 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:15.193521 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:15.193567 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:16:15.193653 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260927 21:16:15.194413 20250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36321
I20260927 21:16:15.194506 20250 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:15.194662 20250 heartbeater.cc:511] Master 127.0.98.62:36321 requested a full tablet report, sending...
I20260927 21:16:15.195122 20061 ts_manager.cc:194] Registered new tserver with Master: a39ad32b567248779e0c46406b0794b1 (127.0.98.1:43615)
I20260927 21:16:15.195750 20061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50701
I20260927 21:16:15.198304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 20122
I20260927 21:16:15.198398 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:16:15.198563 392 external_mini_cluster.cc:1428] Setting key 1045c43c1189cd66eb55eb2af6783acb
I20260927 21:16:15.199586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36321
--builtin_ntp_servers=127.0.98.20:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:16:15.281497 20253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:15.281721 20253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:15.281754 20253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:15.283108 20253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:15.283159 20253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:15.283195 20253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:16:15.284636 20253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:15.284900 20253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:15.285199 20253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:15.290233 20261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:15.290233 20259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:15.290246 20258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:15.290341 20253 server_base.cc:1034] running on GCE node
I20260927 21:16:15.290670 20253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:15.290934 20253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:15.292138 20253 hybrid_clock.cc:648] HybridClock initialized: now 1790543775292110 us; error 48 us; skew 500 ppm
I20260927 21:16:15.293501 20253 webserver.cc:460] Webserver started at http://127.0.98.2:35667/ using document root <none> and password file <none>
I20260927 21:16:15.293738 20253 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:15.293784 20253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:15.293912 20253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:15.295030 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/data/instance:
uuid: "2441fcecbd5544c3a77986a9a76fe19d"
format_stamp: "Formatted at 2026-09-27 21:16:15 on dist-test-slave-r9x1"
server_key: "7618fbbd9184f9ecf7f6c2989e829132"
server_key_iv: "496bc6f1d93da7dc9dd0b6a366cacdf1"
server_key_version: "encryptionkey@0"
I20260927 21:16:15.295368 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2441fcecbd5544c3a77986a9a76fe19d"
format_stamp: "Formatted at 2026-09-27 21:16:15 on dist-test-slave-r9x1"
server_key: "7618fbbd9184f9ecf7f6c2989e829132"
server_key_iv: "496bc6f1d93da7dc9dd0b6a366cacdf1"
server_key_version: "encryptionkey@0"
I20260927 21:16:15.296651 20253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:15.297394 20267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:15.297544 20253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:16:15.297628 20253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/wal
uuid: "2441fcecbd5544c3a77986a9a76fe19d"
format_stamp: "Formatted at 2026-09-27 21:16:15 on dist-test-slave-r9x1"
server_key: "7618fbbd9184f9ecf7f6c2989e829132"
server_key_iv: "496bc6f1d93da7dc9dd0b6a366cacdf1"
server_key_version: "encryptionkey@0"
I20260927 21:16:15.297684 20253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:15.310616 20253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:15.311035 20253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:15.311193 20253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:15.311465 20253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:15.312103 20253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:15.312171 20253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:15.312197 20253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:15.312208 20253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:15.317842 20253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34909
I20260927 21:16:15.317919 20380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34909 every 8 connection(s)
I20260927 21:16:15.318230 20253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:16:15.323624 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.319033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36165 (local address 127.0.98.62:36321)
0927 21:16:15.319093 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.319095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.319280 (+ 185us) server_negotiation.cc:407] Connection header received
0927 21:16:15.319778 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.319779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.319788 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.319829 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:15.320540 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.321064 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.322200 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.322407 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.322466 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.323366 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:15.323367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:15.323368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:15.323371 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:15.323382 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:15.323386 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:15.323415 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.323522 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.323554 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:16:15.323735 20383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.318966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36321 (local address 127.0.98.2:36165)
0927 21:16:15.319162 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:15.319204 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:15.319679 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:15.319891 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:15.319894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:15.319903 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:15.320430 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.320434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.321181 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.321182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.322092 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.322096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.322512 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.322513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.322640 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.323239 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:16:15.323240 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:15.323269 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:16:15.323295 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:15.323423 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:15.323425 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:15.323428 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:16:15.323473 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:16:15.323561 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260927 21:16:15.324193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 20253
I20260927 21:16:15.324219 20381 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36321
I20260927 21:16:15.324299 20381 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:15.324294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:16:15.324497 20381 heartbeater.cc:511] Master 127.0.98.62:36321 requested a full tablet report, sending...
I20260927 21:16:15.324501 392 external_mini_cluster.cc:1428] Setting key 5c32d197bbaed3c6dddce8b2b4a8bb18
I20260927 21:16:15.324932 20062 ts_manager.cc:194] Registered new tserver with Master: 2441fcecbd5544c3a77986a9a76fe19d (127.0.98.2:34909)
I20260927 21:16:15.325381 20062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36165
I20260927 21:16:15.325448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36321
--builtin_ntp_servers=127.0.98.20:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:16:15.410152 20384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:15.410383 20384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:15.410408 20384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:15.411623 20384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:15.411661 20384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:15.411686 20384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:16:15.413060 20384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:15.413309 20384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:15.413619 20384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:15.418941 20392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:15.418977 20390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:15.419030 20384 server_base.cc:1034] running on GCE node
W20260927 21:16:15.418957 20389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:15.419407 20384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:15.419648 20384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:15.420800 20384 hybrid_clock.cc:648] HybridClock initialized: now 1790543775420786 us; error 31 us; skew 500 ppm
I20260927 21:16:15.422201 20384 webserver.cc:460] Webserver started at http://127.0.98.3:39235/ using document root <none> and password file <none>
I20260927 21:16:15.422433 20384 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:15.422487 20384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:15.422613 20384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:15.423733 20384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/data/instance:
uuid: "21d2de00d810405a802292f6ae995e37"
format_stamp: "Formatted at 2026-09-27 21:16:15 on dist-test-slave-r9x1"
server_key: "7b32e960f515e1de34207730fccb322d"
server_key_iv: "8594c9341daa0b584e3acaf5b747459d"
server_key_version: "encryptionkey@0"
I20260927 21:16:15.424075 20384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/wal/instance:
uuid: "21d2de00d810405a802292f6ae995e37"
format_stamp: "Formatted at 2026-09-27 21:16:15 on dist-test-slave-r9x1"
server_key: "7b32e960f515e1de34207730fccb322d"
server_key_iv: "8594c9341daa0b584e3acaf5b747459d"
server_key_version: "encryptionkey@0"
I20260927 21:16:15.425414 20384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:15.426230 20398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:15.426431 20384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:15.426514 20384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/wal
uuid: "21d2de00d810405a802292f6ae995e37"
format_stamp: "Formatted at 2026-09-27 21:16:15 on dist-test-slave-r9x1"
server_key: "7b32e960f515e1de34207730fccb322d"
server_key_iv: "8594c9341daa0b584e3acaf5b747459d"
server_key_version: "encryptionkey@0"
I20260927 21:16:15.426590 20384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:15.448506 20384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:15.448896 20384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:15.449033 20384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:15.449321 20384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:16:15.449695 20384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:16:15.449729 20384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:15.449752 20384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:16:15.449775 20384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:15.455439 20384 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37249
I20260927 21:16:15.455499 20511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37249 every 8 connection(s)
I20260927 21:16:15.455816 20384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:16:15.460405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 20384
I20260927 21:16:15.460532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:16:15.460711 392 external_mini_cluster.cc:1428] Setting key 5118c34adf3fcbf41e0a5d1ad6e11807
I20260927 21:16:15.461402 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.456546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53261 (local address 127.0.98.62:36321)
0927 21:16:15.456617 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.456620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.456838 (+ 218us) server_negotiation.cc:407] Connection header received
0927 21:16:15.457380 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.457381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.457397 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.457452 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:15.458174 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.458672 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.459739 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.459976 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.460040 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.461093 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:15.461095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:15.461095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:15.461101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:15.461116 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:15.461120 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:15.461164 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.461251 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.461305 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:16:15.461645 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.456524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36321 (local address 127.0.98.3:53261)
0927 21:16:15.456713 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:15.456750 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:15.457257 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:15.457520 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:15.457524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:15.457532 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:15.458066 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.458070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.458815 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.458816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.459631 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.459634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.460091 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.460092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.460274 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.460934 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:16:15.460935 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:15.460967 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:16:15.460992 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:15.461174 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:15.461180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:15.461184 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:15.461253 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:16:15.461388 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260927 21:16:15.462080 20512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36321
I20260927 21:16:15.462165 20512 heartbeater.cc:464] Registering TS with master...
I20260927 21:16:15.462342 20512 heartbeater.cc:511] Master 127.0.98.62:36321 requested a full tablet report, sending...
I20260927 21:16:15.462670 20062 ts_manager.cc:194] Registered new tserver with Master: 21d2de00d810405a802292f6ae995e37 (127.0.98.3:37249)
I20260927 21:16:15.463053 20062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53261
I20260927 21:16:15.473035 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:16:15.477409 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.474509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.0.98.62:36321)
0927 21:16:15.474589 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.474591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.474605 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:16:15.474689 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.474690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.474700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.474829 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:15.475032 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.475518 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.476243 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.476428 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.476484 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.477137 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:15.477138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:15.477139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:15.477144 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:15.477160 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:15.477164 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:15.477205 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.477291 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.477318 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:16:15.479355 20062 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:16:15.480943 20062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:16:15.487404 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.483236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.0.98.2:34909)
0927 21:16:15.483423 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.483431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.483497 (+ 66us) server_negotiation.cc:407] Connection header received
0927 21:16:15.483597 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.483600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.483617 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.483693 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:15.483858 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.484763 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.486300 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.486994 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.487061 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.487179 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.487271 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.487311 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260927 21:16:15.487413 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.483042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34909 (local address 127.0.0.1:52922)
0927 21:16:15.483366 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:15.483410 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:15.483493 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:15.483710 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:15.483713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:15.483716 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:15.483789 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.483791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.484851 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.484852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.486211 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.486214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.487081 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.487081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.487170 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.487185 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:16:15.487255 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:16:15.487311 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":60,"thread_start_us":85,"threads_started":1}
I20260927 21:16:15.487632 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.483060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37249 (local address 127.0.0.1:33432)
0927 21:16:15.483465 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:15.483474 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:15.483524 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:15.483811 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:15.483812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:15.483813 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:15.483842 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.483843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.485125 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.485126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.486375 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.486378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.487386 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.487387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.487451 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.487455 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:15.487509 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:16:15.487552 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":55,"thread_start_us":91,"threads_started":1}
I20260927 21:16:15.487761 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.483255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43615 (local address 127.0.0.1:58298)
0927 21:16:15.483566 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:15.483576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:15.483618 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:15.483930 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:15.483931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:15.483932 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:15.483962 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.483963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.485196 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.485197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.486375 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.486378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.487560 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.487561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.487618 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.487622 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:15.487666 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 21:16:15.487703 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260927 21:16:15.488051 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.483403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.0.98.1:43615)
0927 21:16:15.483796 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.483801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.483810 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:15.483844 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.483846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.483861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.483934 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:15.484053 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.485125 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.486475 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.487394 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.487451 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.487529 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.487918 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.487961 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":67,"threads_started":1}
I20260927 21:16:15.488139 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.483180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.0.98.3:37249)
0927 21:16:15.483626 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.483629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.483637 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:15.483666 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.483668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.483682 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.483749 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:15.483893 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.485029 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.486475 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.487193 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.487263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.487347 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.487794 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.488061 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":85,"threads_started":1}
I20260927 21:16:15.488507 20315 tablet_service.cc:1467] Processing CreateTablet for tablet b4fa9bb5fa1f430fa2773afd83afef43 (DEFAULT_TABLE table=test-table [id=1ebb73a7038a4ad3beb21408cbdd6b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:15.488660 20184 tablet_service.cc:1467] Processing CreateTablet for tablet b4fa9bb5fa1f430fa2773afd83afef43 (DEFAULT_TABLE table=test-table [id=1ebb73a7038a4ad3beb21408cbdd6b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:15.488678 20446 tablet_service.cc:1467] Processing CreateTablet for tablet b4fa9bb5fa1f430fa2773afd83afef43 (DEFAULT_TABLE table=test-table [id=1ebb73a7038a4ad3beb21408cbdd6b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:16:15.488935 20315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4fa9bb5fa1f430fa2773afd83afef43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:15.488951 20184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4fa9bb5fa1f430fa2773afd83afef43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:15.489040 20446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4fa9bb5fa1f430fa2773afd83afef43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:15.491853 20526 tablet_bootstrap.cc:492] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: Bootstrap starting.
I20260927 21:16:15.492331 20527 tablet_bootstrap.cc:492] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: Bootstrap starting.
I20260927 21:16:15.492811 20526 tablet_bootstrap.cc:654] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:15.492981 20527 tablet_bootstrap.cc:654] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:15.493144 20526 log.cc:824] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:15.493321 20527 log.cc:824] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:15.493644 20528 tablet_bootstrap.cc:492] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1: Bootstrap starting.
I20260927 21:16:15.494225 20528 tablet_bootstrap.cc:654] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:15.494349 20526 tablet_bootstrap.cc:492] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: No bootstrap required, opened a new log
I20260927 21:16:15.494433 20526 ts_tablet_manager.cc:1397] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:16:15.494536 20528 log.cc:824] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:15.495087 20527 tablet_bootstrap.cc:492] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: No bootstrap required, opened a new log
I20260927 21:16:15.495165 20527 ts_tablet_manager.cc:1397] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:16:15.495260 20528 tablet_bootstrap.cc:492] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1: No bootstrap required, opened a new log
I20260927 21:16:15.495326 20528 ts_tablet_manager.cc:1397] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:16:15.497097 20526 raft_consensus.cc:348] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } }
I20260927 21:16:15.497335 20526 raft_consensus.cc:374] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:15.497383 20526 raft_consensus.cc:729] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d2de00d810405a802292f6ae995e37, State: Initialized, Role: FOLLOWER
I20260927 21:16:15.497310 20528 raft_consensus.cc:348] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } }
I20260927 21:16:15.497488 20528 raft_consensus.cc:374] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:15.497525 20528 raft_consensus.cc:729] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39ad32b567248779e0c46406b0794b1, State: Initialized, Role: FOLLOWER
I20260927 21:16:15.497547 20526 consensus_queue.cc:260] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } }
I20260927 21:16:15.497644 20528 consensus_queue.cc:260] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } }
I20260927 21:16:15.497627 20527 raft_consensus.cc:348] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } }
I20260927 21:16:15.497776 20527 raft_consensus.cc:374] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:15.497816 20527 raft_consensus.cc:729] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2441fcecbd5544c3a77986a9a76fe19d, State: Initialized, Role: FOLLOWER
I20260927 21:16:15.497838 20526 ts_tablet_manager.cc:1428] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 21:16:15.497925 20528 ts_tablet_manager.cc:1428] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:16:15.497948 20512 heartbeater.cc:503] Master 127.0.98.62:36321 was elected leader, sending a full tablet report...
I20260927 21:16:15.497967 20527 consensus_queue.cc:260] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } }
I20260927 21:16:15.498061 20250 heartbeater.cc:503] Master 127.0.98.62:36321 was elected leader, sending a full tablet report...
I20260927 21:16:15.498222 20527 ts_tablet_manager.cc:1428] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:16:15.498294 20381 heartbeater.cc:503] Master 127.0.98.62:36321 was elected leader, sending a full tablet report...
W20260927 21:16:15.568817 20382 tablet.cc:2369] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:16:15.688858 20251 tablet.cc:2369] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:16:15.706406 20513 tablet.cc:2369] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:16:15.868991 20532 raft_consensus.cc:482] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:16:15.869153 20532 raft_consensus.cc:504] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } }
I20260927 21:16:15.869586 20532 leader_election.cc:288] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2441fcecbd5544c3a77986a9a76fe19d (127.0.98.2:34909), a39ad32b567248779e0c46406b0794b1 (127.0.98.1:43615)
I20260927 21:16:15.872710 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.869878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35499 (local address 127.0.98.2:34909)
0927 21:16:15.869932 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.869934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.869945 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:16:15.870020 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.870021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.870032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.870087 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:15.870256 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.870777 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.871855 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.872334 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.872388 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.872397 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.872601 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.872639 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:16:15.872710 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.869811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34909 (local address 127.0.98.3:35499)
0927 21:16:15.869871 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:15.869884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:15.869962 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:15.870117 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:15.870118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:15.870120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:15.870192 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.870194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.870870 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.870872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.871746 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.871749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.872424 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.872425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.872516 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.872520 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:15.872591 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:16:15.872639 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 21:16:15.872872 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.870006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46977 (local address 127.0.98.1:43615)
0927 21:16:15.870068 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.870069 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.870147 (+ 78us) server_negotiation.cc:407] Connection header received
0927 21:16:15.870181 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.870181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.870192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.870248 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:16:15.870437 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.871039 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.872115 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.872534 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.872599 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.872608 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.872758 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.872799 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:16:15.872869 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.869926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43615 (local address 127.0.98.3:46977)
0927 21:16:15.870087 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:15.870095 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:15.870141 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:15.870255 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:15.870256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:15.870257 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:16:15.870327 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.870328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.871172 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.871173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.872008 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:15.872011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.872620 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.872622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:15.872684 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.872688 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:16:15.872742 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:16:15.872784 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260927 21:16:15.873111 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4fa9bb5fa1f430fa2773afd83afef43" candidate_uuid: "21d2de00d810405a802292f6ae995e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2441fcecbd5544c3a77986a9a76fe19d" is_pre_election: true
I20260927 21:16:15.873256 20335 raft_consensus.cc:2454] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d2de00d810405a802292f6ae995e37 in term 0.
I20260927 21:16:15.873515 20399 leader_election.cc:302] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d2de00d810405a802292f6ae995e37, 2441fcecbd5544c3a77986a9a76fe19d; no voters:
I20260927 21:16:15.873556 20204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4fa9bb5fa1f430fa2773afd83afef43" candidate_uuid: "21d2de00d810405a802292f6ae995e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39ad32b567248779e0c46406b0794b1" is_pre_election: true
I20260927 21:16:15.873690 20204 raft_consensus.cc:2454] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d2de00d810405a802292f6ae995e37 in term 0.
I20260927 21:16:15.873688 20532 raft_consensus.cc:2781] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:16:15.873733 20532 raft_consensus.cc:482] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:16:15.873755 20532 raft_consensus.cc:3037] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:15.874536 20532 raft_consensus.cc:504] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } }
I20260927 21:16:15.874645 20532 leader_election.cc:288] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [CANDIDATE]: Term 1 election: Requested vote from peers 2441fcecbd5544c3a77986a9a76fe19d (127.0.98.2:34909), a39ad32b567248779e0c46406b0794b1 (127.0.98.1:43615)
I20260927 21:16:15.874841 20335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4fa9bb5fa1f430fa2773afd83afef43" candidate_uuid: "21d2de00d810405a802292f6ae995e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2441fcecbd5544c3a77986a9a76fe19d"
I20260927 21:16:15.874866 20204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4fa9bb5fa1f430fa2773afd83afef43" candidate_uuid: "21d2de00d810405a802292f6ae995e37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39ad32b567248779e0c46406b0794b1"
I20260927 21:16:15.874931 20204 raft_consensus.cc:3037] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:15.874931 20335 raft_consensus.cc:3037] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:15.875823 20335 raft_consensus.cc:2454] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d2de00d810405a802292f6ae995e37 in term 1.
I20260927 21:16:15.875828 20204 raft_consensus.cc:2454] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d2de00d810405a802292f6ae995e37 in term 1.
I20260927 21:16:15.876034 20399 leader_election.cc:302] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d2de00d810405a802292f6ae995e37, a39ad32b567248779e0c46406b0794b1; no voters:
I20260927 21:16:15.876127 20532 raft_consensus.cc:2781] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:15.876269 20532 raft_consensus.cc:686] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 1 LEADER]: Becoming Leader. State: Replica: 21d2de00d810405a802292f6ae995e37, State: Running, Role: LEADER
I20260927 21:16:15.876358 20532 consensus_queue.cc:237] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } }
I20260927 21:16:15.877169 20059 catalog_manager.cc:5495] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21d2de00d810405a802292f6ae995e37 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "21d2de00d810405a802292f6ae995e37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21d2de00d810405a802292f6ae995e37" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37249 } health_report { overall_health: HEALTHY } } }
I20260927 21:16:15.933525 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.931381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33442 (local address 127.0.98.3:37249)
0927 21:16:15.931422 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:15.931424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:15.931431 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:16:15.931469 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:15.931469 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:15.931486 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:15.931545 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:15.931638 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.932102 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.932821 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:15.933018 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:15.933069 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:15.933186 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:15.933321 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:15.933397 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:15.933436 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 21:16:15.933454 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:16:15.935299 20204 raft_consensus.cc:1261] T b4fa9bb5fa1f430fa2773afd83afef43 P a39ad32b567248779e0c46406b0794b1 [term 1 FOLLOWER]: Refusing update from remote peer 21d2de00d810405a802292f6ae995e37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:15.935297 20335 raft_consensus.cc:1261] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [term 1 FOLLOWER]: Refusing update from remote peer 21d2de00d810405a802292f6ae995e37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:15.935649 20532 consensus_queue.cc:1035] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39ad32b567248779e0c46406b0794b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:15.935806 20536 consensus_queue.cc:1035] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2441fcecbd5544c3a77986a9a76fe19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:15.938557 20540 mvcc.cc:205] Tried to move back new op lower bound from 7334067306228068352 to 7334067305990107136. Current Snapshot: MvccSnapshot[applied={T|T < 7334067306228068352}]
I20260927 21:16:15.938741 20541 mvcc.cc:205] Tried to move back new op lower bound from 7334067306228068352 to 7334067305990107136. Current Snapshot: MvccSnapshot[applied={T|T < 7334067306228068352}]
I20260927 21:16:15.940896 20060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:15.941000 20060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:16:15.941767 20060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4d504f5599a4994be2d44c37dcc54bf: Sending DeleteTablet for 3 replicas of tablet b4fa9bb5fa1f430fa2773afd83afef43
I20260927 21:16:15.942113 20446 tablet_service.cc:1514] Processing DeleteTablet for tablet b4fa9bb5fa1f430fa2773afd83afef43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:15 UTC) from {username='slave'} at 127.0.0.1:33432
I20260927 21:16:15.942361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 20122
I20260927 21:16:15.942498 20315 tablet_service.cc:1514] Processing DeleteTablet for tablet b4fa9bb5fa1f430fa2773afd83afef43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:16:15 UTC) from {username='slave'} at 127.0.0.1:52922
I20260927 21:16:15.943267 20550 tablet_replica.cc:330] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: stopping tablet replica
I20260927 21:16:15.943389 20550 raft_consensus.cc:2229] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:16:15.943522 20550 raft_consensus.cc:2258] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:15.944013 20550 ts_tablet_manager.cc:1905] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:16:15.945863 20550 ts_tablet_manager.cc:1918] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:15.945948 20550 log.cc:1196] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-1/wal/wals/b4fa9bb5fa1f430fa2773afd83afef43
I20260927 21:16:15.946175 20550 ts_tablet_manager.cc:1939] T b4fa9bb5fa1f430fa2773afd83afef43 P 2441fcecbd5544c3a77986a9a76fe19d: Deleting consensus metadata
I20260927 21:16:15.946604 20046 catalog_manager.cc:4841] TS 2441fcecbd5544c3a77986a9a76fe19d (127.0.98.2:34909): tablet b4fa9bb5fa1f430fa2773afd83afef43 (table test-table [id=1ebb73a7038a4ad3beb21408cbdd6b79]) successfully deleted
W20260927 21:16:15.948940 20046 connection.cc:535] client connection to 127.0.98.1:43615 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 21:16:15.949057 20046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:16:15.949199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 20253
I20260927 21:16:15.949211 20549 tablet_replica.cc:330] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: stopping tablet replica
I20260927 21:16:15.949451 20549 raft_consensus.cc:2229] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:16:15.949621 20549 raft_consensus.cc:2258] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:16:15.950549 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:15.950251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43615 (local address 127.0.0.1:58310)
0927 21:16:15.950476 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:15.950515 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260927 21:16:15.950989 20549 ts_tablet_manager.cc:1905] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:16:15.951234 20046 catalog_manager.cc:4584] TS a39ad32b567248779e0c46406b0794b1 (127.0.98.1:43615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4fa9bb5fa1f430fa2773afd83afef43: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43615: connect: Connection refused (error 111)
I20260927 21:16:15.952776 20549 ts_tablet_manager.cc:1918] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:16:15.952880 20549 log.cc:1196] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: Deleting WAL directory at /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543529727471-392-0/minicluster-data/ts-2/wal/wals/b4fa9bb5fa1f430fa2773afd83afef43
I20260927 21:16:15.953137 20549 ts_tablet_manager.cc:1939] T b4fa9bb5fa1f430fa2773afd83afef43 P 21d2de00d810405a802292f6ae995e37: Deleting consensus metadata
I20260927 21:16:15.954767 20049 catalog_manager.cc:4841] TS 21d2de00d810405a802292f6ae995e37 (127.0.98.3:37249): tablet b4fa9bb5fa1f430fa2773afd83afef43 (table test-table [id=1ebb73a7038a4ad3beb21408cbdd6b79]) successfully deleted
I20260927 21:16:15.954849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 20384
I20260927 21:16:15.958983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 20031
2026-09-27T21:16:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1105 ms)
[----------] 6 tests from EncryptionPolicyTest (8743 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:16 dist-test-slave-r9x1 krb5kdc[20554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:16 dist-test-slave-r9x1 krb5kdc[20554](info): set up 2 sockets
Sep 27 21:16:16 dist-test-slave-r9x1 krb5kdc[20554](info): commencing operation
krb5kdc: starting...
W20260927 21:16:16.917098 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:17 dist-test-slave-r9x1 krb5kdc[20554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:17.040301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:17.121670 20567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:17.121910 20567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:17.121946 20567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:17.123243 20567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:17.123302 20567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:17.123322 20567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:17.123338 20567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:17.123353 20567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:17.124977 20567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:17.125244 20567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:17.125494 20567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:17.130267 20573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:17.130252 20575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:17.130285 20567 server_base.cc:1034] running on GCE node
W20260927 21:16:17.130252 20572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:17.130689 20567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:17.131009 20567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:17.132169 20567 hybrid_clock.cc:648] HybridClock initialized: now 1790543777132147 us; error 38 us; skew 500 ppm
Sep 27 21:16:17 dist-test-slave-r9x1 krb5kdc[20554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:17.134764 20567 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:17.135258 20567 webserver.cc:460] Webserver started at http://127.0.0.1:45471/ using document root <none> and password file <none>
I20260927 21:16:17.135456 20567 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:17.135495 20567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:17.135591 20567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:17.136615 20567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "dfc3ede82fb2482b8527ec7a2c731967"
format_stamp: "Formatted at 2026-09-27 21:16:17 on dist-test-slave-r9x1"
server_key: "2d7f48c83aab0c681e99bbee89d1b411"
server_key_iv: "d007a3d8fdf45364c203d41443c147f6"
server_key_version: "encryptionkey@0"
I20260927 21:16:17.136938 20567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "dfc3ede82fb2482b8527ec7a2c731967"
format_stamp: "Formatted at 2026-09-27 21:16:17 on dist-test-slave-r9x1"
server_key: "2d7f48c83aab0c681e99bbee89d1b411"
server_key_iv: "d007a3d8fdf45364c203d41443c147f6"
server_key_version: "encryptionkey@0"
I20260927 21:16:17.138263 20567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:17.138955 20582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:17.139122 20567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:16:17.139204 20567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "dfc3ede82fb2482b8527ec7a2c731967"
format_stamp: "Formatted at 2026-09-27 21:16:17 on dist-test-slave-r9x1"
server_key: "2d7f48c83aab0c681e99bbee89d1b411"
server_key_iv: "d007a3d8fdf45364c203d41443c147f6"
server_key_version: "encryptionkey@0"
I20260927 21:16:17.139261 20567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:17.150486 20567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:17.152567 20567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:17.152686 20567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:17.156363 20567 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45257
I20260927 21:16:17.156418 20634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45257 every 8 connection(s)
I20260927 21:16:17.156735 20567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:17.157627 20635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:17.159888 20635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967: Bootstrap starting.
I20260927 21:16:17.160434 20635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:17.160671 20635 log.cc:824] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:17.161412 20635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967: No bootstrap required, opened a new log
I20260927 21:16:17.162998 20635 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc3ede82fb2482b8527ec7a2c731967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260927 21:16:17.163146 20635 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:17.163167 20635 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc3ede82fb2482b8527ec7a2c731967, State: Initialized, Role: FOLLOWER
I20260927 21:16:17.163275 20635 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc3ede82fb2482b8527ec7a2c731967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260927 21:16:17.163329 20635 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:17.163357 20635 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:17.163396 20635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:17.163945 20635 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc3ede82fb2482b8527ec7a2c731967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260927 21:16:17.164125 20635 leader_election.cc:302] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfc3ede82fb2482b8527ec7a2c731967; no voters:
I20260927 21:16:17.164331 20635 leader_election.cc:288] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:17.164391 20638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:17.164543 20638 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [term 1 LEADER]: Becoming Leader. State: Replica: dfc3ede82fb2482b8527ec7a2c731967, State: Running, Role: LEADER
I20260927 21:16:17.164592 20635 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:17.164642 20638 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc3ede82fb2482b8527ec7a2c731967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260927 21:16:17.165016 20640 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfc3ede82fb2482b8527ec7a2c731967. Latest consensus state: current_term: 1 leader_uuid: "dfc3ede82fb2482b8527ec7a2c731967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc3ede82fb2482b8527ec7a2c731967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } } }
I20260927 21:16:17.165092 20640 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:17.165096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 20567
I20260927 21:16:17.165201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:17.165153 20639 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfc3ede82fb2482b8527ec7a2c731967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc3ede82fb2482b8527ec7a2c731967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } } }
I20260927 21:16:17.165228 20639 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:17.165427 392 external_mini_cluster.cc:1428] Setting key 075562e21081264234b391c4a3fb9e3b
I20260927 21:16:17.165673 20646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:17.166213 20646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:17.167883 20646 catalog_manager.cc:1347] Generated new cluster ID: 10ed89724d344d92abc5dda5db5c1eb8
I20260927 21:16:17.167936 20646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:16:17 dist-test-slave-r9x1 krb5kdc[20554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:16:17.178222 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:17.166115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42677 (local address 127.0.0.1:45257)
0927 21:16:17.166402 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:17.166413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:17.166531 (+ 118us) server_negotiation.cc:407] Connection header received
0927 21:16:17.166648 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:17.166655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:17.166712 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:17.166855 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:17.167604 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:17.168988 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:17.170102 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:17.170745 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:17.170840 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:17.173987 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:17.173994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:17.173994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:17.174010 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:17.176048 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:17.176556 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:17.176560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:17.176560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:17.176681 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:17.176888 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:17.176889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:17.176889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:17.177235 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:17.177436 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:17.177737 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 21:16:17.177965 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":59,"threads_started":1}
I20260927 21:16:17.179180 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:17.188496 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:17.180963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47483 (local address 127.0.0.1:45257)
0927 21:16:17.181001 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:17.181002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:17.181011 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:17.181084 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:17.181085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:17.181094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:17.181141 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:17.181855 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:17.182628 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:17.183701 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:17.184012 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:17.184069 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:17.185623 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:17.185624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:17.185625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:17.185633 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:17.187152 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:17.187609 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:17.187610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:17.187610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:17.187666 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:17.187875 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:17.187875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:17.187876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:17.187996 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:17.188086 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:17.188224 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:16:17.188284 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:16:17.195084 20646 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:17.195662 20646 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:17.210444 20646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfc3ede82fb2482b8527ec7a2c731967: Generated new TSK 0
I20260927 21:16:17.210645 20646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:17.215281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 20567
W20260927 21:16:17.219321 20552 connection.cc:535] client connection to 127.0.0.1:45257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45257): Connection reset by peer (error 104)
2026-09-27T21:16:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1255 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:17 dist-test-slave-r9x1 krb5kdc[20666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:17 dist-test-slave-r9x1 krb5kdc[20666](info): set up 2 sockets
Sep 27 21:16:17 dist-test-slave-r9x1 krb5kdc[20666](info): commencing operation
krb5kdc: starting...
W20260927 21:16:18.173359 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:18 dist-test-slave-r9x1 krb5kdc[20666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:18.293642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:18.374151 20679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:18.374392 20679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:18.374418 20679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:18.375641 20679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:18.375687 20679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:18.375701 20679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:18.375713 20679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:18.375723 20679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:18.377377 20679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:18.377620 20679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:18.377820 20679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:18.382934 20687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:18.382934 20685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:18.383019 20684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:18.383159 20679 server_base.cc:1034] running on GCE node
I20260927 21:16:18.383402 20679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:18.383694 20679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:18.384845 20679 hybrid_clock.cc:648] HybridClock initialized: now 1790543778384832 us; error 29 us; skew 500 ppm
Sep 27 21:16:18 dist-test-slave-r9x1 krb5kdc[20666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:18.393139 20679 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:18.393710 20679 webserver.cc:460] Webserver started at http://127.0.0.1:42129/ using document root <none> and password file <none>
I20260927 21:16:18.393926 20679 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:18.393980 20679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:18.394111 20679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:18.395220 20679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "03e2f37d962341c5b2cbee33a36d75d8"
format_stamp: "Formatted at 2026-09-27 21:16:18 on dist-test-slave-r9x1"
server_key: "3c5d78b741fbac372fbcc1ae6c3c86a2"
server_key_iv: "6c83c57c9e64d1c8aea256635e68665f"
server_key_version: "encryptionkey@0"
I20260927 21:16:18.395576 20679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "03e2f37d962341c5b2cbee33a36d75d8"
format_stamp: "Formatted at 2026-09-27 21:16:18 on dist-test-slave-r9x1"
server_key: "3c5d78b741fbac372fbcc1ae6c3c86a2"
server_key_iv: "6c83c57c9e64d1c8aea256635e68665f"
server_key_version: "encryptionkey@0"
I20260927 21:16:18.397001 20679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:16:18.397878 20694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:18.398070 20679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:16:18.398159 20679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "03e2f37d962341c5b2cbee33a36d75d8"
format_stamp: "Formatted at 2026-09-27 21:16:18 on dist-test-slave-r9x1"
server_key: "3c5d78b741fbac372fbcc1ae6c3c86a2"
server_key_iv: "6c83c57c9e64d1c8aea256635e68665f"
server_key_version: "encryptionkey@0"
I20260927 21:16:18.398238 20679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:18.418903 20679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:18.420990 20679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:18.421142 20679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:18.425073 20679 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37467
I20260927 21:16:18.425112 20746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37467 every 8 connection(s)
I20260927 21:16:18.425485 20679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:18.426321 20747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:18.428583 20747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8: Bootstrap starting.
I20260927 21:16:18.428746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 20679
I20260927 21:16:18.428858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:18.429029 392 external_mini_cluster.cc:1428] Setting key 1677529d6bd1861d0596eb844616ac88
I20260927 21:16:18.429126 20747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:18.429426 20747 log.cc:824] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:18.430112 20747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8: No bootstrap required, opened a new log
I20260927 21:16:18.431516 20747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e2f37d962341c5b2cbee33a36d75d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260927 21:16:18.431663 20747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:18.431695 20747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e2f37d962341c5b2cbee33a36d75d8, State: Initialized, Role: FOLLOWER
I20260927 21:16:18.431834 20747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e2f37d962341c5b2cbee33a36d75d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260927 21:16:18.431906 20747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:18.431927 20747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:18.431963 20747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:18.432526 20747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e2f37d962341c5b2cbee33a36d75d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260927 21:16:18.432626 20747 leader_election.cc:302] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03e2f37d962341c5b2cbee33a36d75d8; no voters:
I20260927 21:16:18.432797 20747 leader_election.cc:288] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:18.432847 20752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:18.432961 20752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [term 1 LEADER]: Becoming Leader. State: Replica: 03e2f37d962341c5b2cbee33a36d75d8, State: Running, Role: LEADER
I20260927 21:16:18.433054 20747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:18.433048 20752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e2f37d962341c5b2cbee33a36d75d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260927 21:16:18.433429 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e2f37d962341c5b2cbee33a36d75d8. Latest consensus state: current_term: 1 leader_uuid: "03e2f37d962341c5b2cbee33a36d75d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e2f37d962341c5b2cbee33a36d75d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260927 21:16:18.433499 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:18.433630 20753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03e2f37d962341c5b2cbee33a36d75d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e2f37d962341c5b2cbee33a36d75d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260927 21:16:18.433689 20753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:18.433811 20757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:18.434262 20757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:18.435164 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:18.429567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36521 (local address 127.0.0.1:37467)
0927 21:16:18.429732 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:18.429741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:18.429769 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:16:18.430211 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:18.430217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:18.430264 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:18.430394 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:18.430620 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:18.432052 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:18.432892 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:18.433532 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:18.434011 (+ 479us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:18.434393 (+ 382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:18.434396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:18.434397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:18.434657 (+ 260us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:18.434688 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:18.434699 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:18.434762 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:18.434933 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:16:18.434966 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260927 21:16:18.435932 20757 catalog_manager.cc:1347] Generated new cluster ID: 26a5eae8f3424222a9f06ec483b29298
I20260927 21:16:18.435987 20757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:18.436132 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:18.441308 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:18.437570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60161 (local address 127.0.0.1:37467)
0927 21:16:18.437606 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:18.437607 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:18.437744 (+ 137us) server_negotiation.cc:407] Connection header received
0927 21:16:18.437791 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:18.437792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:18.437814 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:18.437867 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:18.437993 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:18.438741 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:18.439735 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:18.440164 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:18.440250 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:18.440986 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:18.440987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:18.440988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:18.440993 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:18.441040 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:18.441046 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:18.441139 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:18.441196 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 21:16:18.441210 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:16:18.460491 20757 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:18.461050 20757 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:18.468624 20757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03e2f37d962341c5b2cbee33a36d75d8: Generated new TSK 0
I20260927 21:16:18.468811 20757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:18.475983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 20679
W20260927 21:16:18.479851 20664 connection.cc:535] client connection to 127.0.0.1:37467 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37467): Connection reset by peer (error 104)
2026-09-27T21:16:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1260 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:18 dist-test-slave-r9x1 krb5kdc[20778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:18 dist-test-slave-r9x1 krb5kdc[20778](info): set up 2 sockets
Sep 27 21:16:18 dist-test-slave-r9x1 krb5kdc[20778](info): commencing operation
krb5kdc: starting...
W20260927 21:16:19.441069 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:19 dist-test-slave-r9x1 krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:19.565410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:19.646842 20791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:19.647079 20791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:19.647112 20791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:19.648373 20791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:19.648429 20791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:19.648448 20791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:19.648464 20791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:19.648480 20791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:19.650104 20791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:19.650373 20791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:19.650588 20791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:19.655902 20796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:19.655993 20791 server_base.cc:1034] running on GCE node
W20260927 21:16:19.655920 20797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:19.655923 20799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:19.656471 20791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:19.656850 20791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:19.658041 20791 hybrid_clock.cc:648] HybridClock initialized: now 1790543779658018 us; error 37 us; skew 500 ppm
Sep 27 21:16:19 dist-test-slave-r9x1 krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:19.660784 20791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:19.661278 20791 webserver.cc:460] Webserver started at http://127.0.0.1:34031/ using document root <none> and password file <none>
I20260927 21:16:19.661528 20791 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:19.661577 20791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:19.661705 20791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:19.662801 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "c4f673ece00045f888611f101fce84ec"
format_stamp: "Formatted at 2026-09-27 21:16:19 on dist-test-slave-r9x1"
server_key: "79db229641694efda582e5c13d77a21c"
server_key_iv: "c9d99c1a1c3ee0eecbee7fb58c0728d9"
server_key_version: "encryptionkey@0"
I20260927 21:16:19.663127 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4f673ece00045f888611f101fce84ec"
format_stamp: "Formatted at 2026-09-27 21:16:19 on dist-test-slave-r9x1"
server_key: "79db229641694efda582e5c13d77a21c"
server_key_iv: "c9d99c1a1c3ee0eecbee7fb58c0728d9"
server_key_version: "encryptionkey@0"
I20260927 21:16:19.664453 20791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:19.665195 20806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:19.665362 20791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:19.665457 20791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "c4f673ece00045f888611f101fce84ec"
format_stamp: "Formatted at 2026-09-27 21:16:19 on dist-test-slave-r9x1"
server_key: "79db229641694efda582e5c13d77a21c"
server_key_iv: "c9d99c1a1c3ee0eecbee7fb58c0728d9"
server_key_version: "encryptionkey@0"
I20260927 21:16:19.665525 20791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:19.668838 20791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:19.670792 20791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:19.670935 20791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:19.674907 20791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44595
I20260927 21:16:19.674942 20858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44595 every 8 connection(s)
I20260927 21:16:19.675285 20791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:19.676251 20859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:19.678709 20859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec: Bootstrap starting.
I20260927 21:16:19.679270 20859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:19.679522 20859 log.cc:824] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:19.680205 20859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec: No bootstrap required, opened a new log
I20260927 21:16:19.680444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 20791
I20260927 21:16:19.680557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:19.680742 392 external_mini_cluster.cc:1428] Setting key 53f108bc6b4364d78fa8cfeb175d8836
I20260927 21:16:19.682305 20859 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f673ece00045f888611f101fce84ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260927 21:16:19.682483 20859 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:19.682523 20859 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f673ece00045f888611f101fce84ec, State: Initialized, Role: FOLLOWER
I20260927 21:16:19.682703 20859 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f673ece00045f888611f101fce84ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260927 21:16:19.682811 20859 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:19.682835 20859 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:19.682873 20859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:19.683607 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:19.681284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38285 (local address 127.0.0.1:44595)
0927 21:16:19.681458 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:19.681466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:19.681491 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:16:19.682279 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:19.682284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:19.682311 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:19.682431 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:19.683139 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:19.683142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:19.683143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:19.683146 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:19.683188 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:19.683200 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:19.683265 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:19.683343 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 21:16:19.683379 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260927 21:16:19.683975 20859 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f673ece00045f888611f101fce84ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260927 21:16:19.684080 20859 leader_election.cc:302] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4f673ece00045f888611f101fce84ec; no voters:
I20260927 21:16:19.684266 20859 leader_election.cc:288] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:19.684307 20864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:19.684450 20864 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [term 1 LEADER]: Becoming Leader. State: Replica: c4f673ece00045f888611f101fce84ec, State: Running, Role: LEADER
I20260927 21:16:19.684448 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:19.684543 20859 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:19.684593 20864 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f673ece00045f888611f101fce84ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260927 21:16:19.684928 20864 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4f673ece00045f888611f101fce84ec. Latest consensus state: current_term: 1 leader_uuid: "c4f673ece00045f888611f101fce84ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f673ece00045f888611f101fce84ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } } }
I20260927 21:16:19.685024 20864 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:19.685170 20866 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4f673ece00045f888611f101fce84ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f673ece00045f888611f101fce84ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } } }
I20260927 21:16:19.685230 20866 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:19.685381 20871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:19.685902 20871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:19.687413 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:19.686116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58195 (local address 127.0.0.1:44595)
0927 21:16:19.686183 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:19.686185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:19.686508 (+ 323us) server_negotiation.cc:407] Connection header received
0927 21:16:19.686577 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:19.686578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:19.686585 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:19.686629 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:19.687195 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:19.687196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:19.687196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:19.687201 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:19.687217 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:19.687222 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:19.687289 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:19.687341 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:16:19.687355 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:16:19.687625 20871 catalog_manager.cc:1347] Generated new cluster ID: 1cb0b46e7433452ca720fe8f2a2f11f3
I20260927 21:16:19.687670 20871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:19.714608 20871 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:19.714847 20871 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:19.721998 20871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4f673ece00045f888611f101fce84ec: Generated new TSK 0
I20260927 21:16:19.722205 20871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:19.730168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 20791
2026-09-27T21:16:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:19 dist-test-slave-r9x1 krb5kdc[20890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:19 dist-test-slave-r9x1 krb5kdc[20890](info): set up 2 sockets
Sep 27 21:16:19 dist-test-slave-r9x1 krb5kdc[20890](info): commencing operation
krb5kdc: starting...
W20260927 21:16:20.689069 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:20 dist-test-slave-r9x1 krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:20.804769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:20.885437 20903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:20.885660 20903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:20.885684 20903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:20.886895 20903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:20.886940 20903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:20.886952 20903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:20.886965 20903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:20.886974 20903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:20.888514 20903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:20.888736 20903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:20.888921 20903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:20.893769 20911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:20.893754 20908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:20.893756 20909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:20.893810 20903 server_base.cc:1034] running on GCE node
I20260927 21:16:20.894145 20903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:20.894402 20903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:20.895537 20903 hybrid_clock.cc:648] HybridClock initialized: now 1790543780895524 us; error 29 us; skew 500 ppm
Sep 27 21:16:20 dist-test-slave-r9x1 krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:20.897991 20903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:20.898422 20903 webserver.cc:460] Webserver started at http://127.0.0.1:45545/ using document root <none> and password file <none>
I20260927 21:16:20.898605 20903 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:20.898643 20903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:20.898734 20903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:20.899777 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "f9f4cb44ca9d4db3afc700ae034342b4"
format_stamp: "Formatted at 2026-09-27 21:16:20 on dist-test-slave-r9x1"
server_key: "a31e5314dcf32bfe7f460fe7c47e76db"
server_key_iv: "5622c200a3b3d52b0e643520297a40f1"
server_key_version: "encryptionkey@0"
I20260927 21:16:20.900094 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9f4cb44ca9d4db3afc700ae034342b4"
format_stamp: "Formatted at 2026-09-27 21:16:20 on dist-test-slave-r9x1"
server_key: "a31e5314dcf32bfe7f460fe7c47e76db"
server_key_iv: "5622c200a3b3d52b0e643520297a40f1"
server_key_version: "encryptionkey@0"
I20260927 21:16:20.901391 20903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:20.902158 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:20.902374 20903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:20.902463 20903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "f9f4cb44ca9d4db3afc700ae034342b4"
format_stamp: "Formatted at 2026-09-27 21:16:20 on dist-test-slave-r9x1"
server_key: "a31e5314dcf32bfe7f460fe7c47e76db"
server_key_iv: "5622c200a3b3d52b0e643520297a40f1"
server_key_version: "encryptionkey@0"
I20260927 21:16:20.902529 20903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:20.927721 20903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:20.929943 20903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:20.930063 20903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:20.933831 20903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42705
I20260927 21:16:20.933883 20970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42705 every 8 connection(s)
I20260927 21:16:20.934203 20903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:20.935055 20971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:20.938055 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4: Bootstrap starting.
I20260927 21:16:20.938710 20971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:20.938946 20971 log.cc:824] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:20.939602 20971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4: No bootstrap required, opened a new log
I20260927 21:16:20.939755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 20903
I20260927 21:16:20.939869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:20.940037 392 external_mini_cluster.cc:1428] Setting key 8934793ef6d901d4556c25cdee545cf1
I20260927 21:16:20.941570 20971 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f4cb44ca9d4db3afc700ae034342b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260927 21:16:20.941711 20971 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:20.941733 20971 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f4cb44ca9d4db3afc700ae034342b4, State: Initialized, Role: FOLLOWER
I20260927 21:16:20.941876 20971 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f4cb44ca9d4db3afc700ae034342b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260927 21:16:20.941962 20971 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:20.942001 20971 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:20.942049 20971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:20.943063 20971 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f4cb44ca9d4db3afc700ae034342b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260927 21:16:20.943154 20971 leader_election.cc:302] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9f4cb44ca9d4db3afc700ae034342b4; no voters:
I20260927 21:16:20.943320 20971 leader_election.cc:288] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:20.943365 20976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:20.943542 20976 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [term 1 LEADER]: Becoming Leader. State: Replica: f9f4cb44ca9d4db3afc700ae034342b4, State: Running, Role: LEADER
I20260927 21:16:20.943570 20971 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:20.943632 20976 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f4cb44ca9d4db3afc700ae034342b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } }
I20260927 21:16:20.943957 20976 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9f4cb44ca9d4db3afc700ae034342b4. Latest consensus state: current_term: 1 leader_uuid: "f9f4cb44ca9d4db3afc700ae034342b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f4cb44ca9d4db3afc700ae034342b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } } }
I20260927 21:16:20.944020 20976 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:20.944159 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9f4cb44ca9d4db3afc700ae034342b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f4cb44ca9d4db3afc700ae034342b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42705 } } }
I20260927 21:16:20.944204 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:20.944300 20983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:20.944991 20983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:20.946967 20983 catalog_manager.cc:1347] Generated new cluster ID: f0d53774918e403f9e82373acfbc5052
I20260927 21:16:20.947026 20983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:16:20 dist-test-slave-r9x1 krb5kdc[20890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:16:20.950884 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:20.940527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57371 (local address 127.0.0.1:42705)
0927 21:16:20.940720 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:20.940730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:20.940760 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:16:20.941571 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:20.941580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:20.941636 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:20.941802 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:20.942534 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:20.943650 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:20.944765 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:20.945416 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:20.945518 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:20.947800 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:20.947805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:20.947805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:20.947818 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:20.949422 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:20.949747 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:20.949750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:20.949751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:20.949867 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:20.950048 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:20.950049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:20.950050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:20.950286 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:20.950431 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:20.950532 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 21:16:20.950670 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260927 21:16:20.951754 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:20.959537 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:20.953569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44289 (local address 127.0.0.1:42705)
0927 21:16:20.953605 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:20.953608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:20.953808 (+ 200us) server_negotiation.cc:407] Connection header received
0927 21:16:20.953894 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:20.953895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:20.953906 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:20.953947 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:20.954438 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:20.954919 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:20.955591 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:20.955810 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:20.955867 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:20.957032 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:20.957034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:20.957034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:20.957041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:20.958511 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:20.958838 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:20.958839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:20.958840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:20.958894 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:20.959061 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:20.959062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:20.959062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:20.959173 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:20.959257 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:20.959362 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:16:20.959443 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:16:20.966411 20983 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:20.966979 20983 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:20.975924 20983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9f4cb44ca9d4db3afc700ae034342b4: Generated new TSK 0
I20260927 21:16:20.976130 20983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:20.978008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 20903
2026-09-27T21:16:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:21 dist-test-slave-r9x1 krb5kdc[21002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:21 dist-test-slave-r9x1 krb5kdc[21002](info): set up 2 sockets
Sep 27 21:16:21 dist-test-slave-r9x1 krb5kdc[21002](info): commencing operation
krb5kdc: starting...
W20260927 21:16:21.933033 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:22 dist-test-slave-r9x1 krb5kdc[21002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:22.064653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:22.147094 21015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:22.147334 21015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:22.147358 21015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:22.148641 21015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:22.148706 21015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:22.148720 21015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:22.148731 21015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:22.148742 21015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:22.150466 21015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:22.150753 21015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:22.150993 21015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:22.156181 21023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:22.156170 21020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:22.156170 21021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:22.156514 21015 server_base.cc:1034] running on GCE node
I20260927 21:16:22.156767 21015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:22.157078 21015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:22.158233 21015 hybrid_clock.cc:648] HybridClock initialized: now 1790543782158190 us; error 61 us; skew 500 ppm
Sep 27 21:16:22 dist-test-slave-r9x1 krb5kdc[21002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:22.160993 21015 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:22.161486 21015 webserver.cc:460] Webserver started at http://127.0.0.1:43705/ using document root <none> and password file <none>
I20260927 21:16:22.161687 21015 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:22.161736 21015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:22.161827 21015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:22.162909 21015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "8b8134b2b1604b22889867098c497404"
format_stamp: "Formatted at 2026-09-27 21:16:22 on dist-test-slave-r9x1"
server_key: "56f27601ebbbefcd3071300104d6cd4d"
server_key_iv: "f05c4397d69f47d56e90d66fd6ded9d2"
server_key_version: "encryptionkey@0"
I20260927 21:16:22.163218 21015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b8134b2b1604b22889867098c497404"
format_stamp: "Formatted at 2026-09-27 21:16:22 on dist-test-slave-r9x1"
server_key: "56f27601ebbbefcd3071300104d6cd4d"
server_key_iv: "f05c4397d69f47d56e90d66fd6ded9d2"
server_key_version: "encryptionkey@0"
I20260927 21:16:22.165027 21015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:16:22.165905 21030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:22.166054 21015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:16:22.166137 21015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "8b8134b2b1604b22889867098c497404"
format_stamp: "Formatted at 2026-09-27 21:16:22 on dist-test-slave-r9x1"
server_key: "56f27601ebbbefcd3071300104d6cd4d"
server_key_iv: "f05c4397d69f47d56e90d66fd6ded9d2"
server_key_version: "encryptionkey@0"
I20260927 21:16:22.166213 21015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:22.184769 21015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:22.186908 21015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:22.187028 21015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:22.190807 21015 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35331
I20260927 21:16:22.190915 21082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35331 every 8 connection(s)
I20260927 21:16:22.191756 21015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:22.192723 21083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:22.195070 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404: Bootstrap starting.
I20260927 21:16:22.195619 21083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:22.195880 21083 log.cc:824] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:22.196566 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404: No bootstrap required, opened a new log
I20260927 21:16:22.198248 21083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8134b2b1604b22889867098c497404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35331 } }
I20260927 21:16:22.198398 21083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:22.198432 21083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8134b2b1604b22889867098c497404, State: Initialized, Role: FOLLOWER
I20260927 21:16:22.198567 21083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8134b2b1604b22889867098c497404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35331 } }
I20260927 21:16:22.198632 21083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:22.198665 21083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:22.198719 21083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:22.199275 21083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8134b2b1604b22889867098c497404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35331 } }
I20260927 21:16:22.199379 21083 leader_election.cc:302] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8134b2b1604b22889867098c497404; no voters:
I20260927 21:16:22.199548 21083 leader_election.cc:288] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:22.199663 21086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:22.199821 21083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:22.199980 21086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [term 1 LEADER]: Becoming Leader. State: Replica: 8b8134b2b1604b22889867098c497404, State: Running, Role: LEADER
I20260927 21:16:22.200114 21086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8134b2b1604b22889867098c497404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35331 } }
I20260927 21:16:22.200168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 21015
I20260927 21:16:22.200284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:22.200654 392 external_mini_cluster.cc:1428] Setting key 7cd85c2bc191c5e71a5b1a2b2efce767
I20260927 21:16:22.200883 21087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b8134b2b1604b22889867098c497404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8134b2b1604b22889867098c497404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35331 } } }
I20260927 21:16:22.200948 21088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8134b2b1604b22889867098c497404. Latest consensus state: current_term: 1 leader_uuid: "8b8134b2b1604b22889867098c497404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8134b2b1604b22889867098c497404" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35331 } } }
I20260927 21:16:22.200987 21087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:22.201023 21088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:22.201478 21097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:22.202296 21097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:22.204026 21097 catalog_manager.cc:1347] Generated new cluster ID: 66d802f5e4a148fdbfcc3968f7e132cc
I20260927 21:16:22.204094 21097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:22.206429 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:22.201541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45347 (local address 127.0.0.1:35331)
0927 21:16:22.201726 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:22.201736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:22.201897 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:16:22.202080 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:22.202086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:22.202133 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:22.202262 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:22.202485 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:22.203643 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:22.204545 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:22.204974 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:22.205071 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:22.205808 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:22.205813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:22.205813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:22.205986 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:22.206019 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:22.206032 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:22.206103 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:22.206175 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 21:16:22.206242 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260927 21:16:22.207388 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:22.216254 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:22.208941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45591 (local address 127.0.0.1:35331)
0927 21:16:22.209076 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:22.209078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:22.213570 (+ 4492us) server_negotiation.cc:407] Connection header received
0927 21:16:22.213644 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:22.213646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:22.213661 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:22.213716 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:22.213845 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:22.214362 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:22.215111 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:22.215307 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:22.215363 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:22.216017 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:22.216018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:22.216018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:22.216024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:22.216039 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:22.216043 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:22.216088 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:22.216136 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 21:16:22.216166 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 21:16:22.221828 21097 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:22.222368 21097 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:22.226679 21097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b8134b2b1604b22889867098c497404: Generated new TSK 0
I20260927 21:16:22.226903 21097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:22.231292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 21015
2026-09-27T21:16:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:22 dist-test-slave-r9x1 krb5kdc[21114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:22 dist-test-slave-r9x1 krb5kdc[21114](info): set up 2 sockets
Sep 27 21:16:22 dist-test-slave-r9x1 krb5kdc[21114](info): commencing operation
krb5kdc: starting...
W20260927 21:16:23.193181 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:23 dist-test-slave-r9x1 krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:23.313499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:23.397691 21127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:23.397939 21127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:23.397974 21127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:23.399267 21127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:23.399325 21127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:23.399344 21127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:23.399362 21127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:23.399377 21127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:23.401036 21127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:23.401345 21127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:23.401572 21127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:23.406417 21132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:23.406412 21135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:23.406447 21127 server_base.cc:1034] running on GCE node
W20260927 21:16:23.406440 21133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:23.406867 21127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:23.407162 21127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:23.408329 21127 hybrid_clock.cc:648] HybridClock initialized: now 1790543783408307 us; error 32 us; skew 500 ppm
Sep 27 21:16:23 dist-test-slave-r9x1 krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:23.416807 21127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:23.417361 21127 webserver.cc:460] Webserver started at http://127.0.0.1:45101/ using document root <none> and password file <none>
I20260927 21:16:23.417647 21127 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:23.417759 21127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:23.417898 21127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:23.418979 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "7ad03f8e2b154c61aae6fd90e5701228"
format_stamp: "Formatted at 2026-09-27 21:16:23 on dist-test-slave-r9x1"
server_key: "d1698f1b121527bafe01812735649125"
server_key_iv: "80aa2bacf8fd4ec1f799a9dbab868111"
server_key_version: "encryptionkey@0"
I20260927 21:16:23.419332 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ad03f8e2b154c61aae6fd90e5701228"
format_stamp: "Formatted at 2026-09-27 21:16:23 on dist-test-slave-r9x1"
server_key: "d1698f1b121527bafe01812735649125"
server_key_iv: "80aa2bacf8fd4ec1f799a9dbab868111"
server_key_version: "encryptionkey@0"
I20260927 21:16:23.420692 21127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:23.421584 21142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:23.421778 21127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:23.421877 21127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "7ad03f8e2b154c61aae6fd90e5701228"
format_stamp: "Formatted at 2026-09-27 21:16:23 on dist-test-slave-r9x1"
server_key: "d1698f1b121527bafe01812735649125"
server_key_iv: "80aa2bacf8fd4ec1f799a9dbab868111"
server_key_version: "encryptionkey@0"
I20260927 21:16:23.421945 21127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:23.425523 21127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:23.427556 21127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:23.427699 21127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:23.431456 21127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45073
I20260927 21:16:23.431496 21194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45073 every 8 connection(s)
I20260927 21:16:23.431816 21127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:23.432745 21195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:23.435079 21195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228: Bootstrap starting.
I20260927 21:16:23.435681 21195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:23.435945 21195 log.cc:824] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:23.436708 21195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228: No bootstrap required, opened a new log
I20260927 21:16:23.438509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 21127
I20260927 21:16:23.438630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:23.438485 21195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03f8e2b154c61aae6fd90e5701228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45073 } }
I20260927 21:16:23.438690 21195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:23.438735 21195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad03f8e2b154c61aae6fd90e5701228, State: Initialized, Role: FOLLOWER
I20260927 21:16:23.438827 392 external_mini_cluster.cc:1428] Setting key fb43a531383f0d90d42bab0d1f4ebb0f
I20260927 21:16:23.438885 21195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03f8e2b154c61aae6fd90e5701228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45073 } }
I20260927 21:16:23.438961 21195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:23.438990 21195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:23.439033 21195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:23.439761 21195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03f8e2b154c61aae6fd90e5701228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45073 } }
I20260927 21:16:23.439869 21195 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ad03f8e2b154c61aae6fd90e5701228; no voters:
I20260927 21:16:23.440033 21195 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:23.440060 21200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:23.440227 21200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [term 1 LEADER]: Becoming Leader. State: Replica: 7ad03f8e2b154c61aae6fd90e5701228, State: Running, Role: LEADER
I20260927 21:16:23.440294 21195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:23.440323 21200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03f8e2b154c61aae6fd90e5701228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45073 } }
I20260927 21:16:23.440668 21200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad03f8e2b154c61aae6fd90e5701228. Latest consensus state: current_term: 1 leader_uuid: "7ad03f8e2b154c61aae6fd90e5701228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03f8e2b154c61aae6fd90e5701228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45073 } } }
I20260927 21:16:23.440737 21200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:23.440901 21201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ad03f8e2b154c61aae6fd90e5701228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad03f8e2b154c61aae6fd90e5701228" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45073 } } }
I20260927 21:16:23.440971 21201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:23.441093 21206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:23.441644 21206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:23.441672 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:23.439396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42117 (local address 127.0.0.1:45073)
0927 21:16:23.439919 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:23.439929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:23.439965 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:16:23.440093 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:23.440101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:23.440135 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:23.440569 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:23.441048 (+ 479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:23.441052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:23.441052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:23.441056 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:23.441099 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:23.441115 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:23.441424 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:23.441458 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 21:16:23.441501 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":415,"threads_started":1}
I20260927 21:16:23.443203 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:23.443610 21206 catalog_manager.cc:1347] Generated new cluster ID: 8a247a72b7b04226b9b58653c637c347
I20260927 21:16:23.443709 21206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:23.450538 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:23.445755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41855 (local address 127.0.0.1:45073)
0927 21:16:23.445813 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:23.445815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:23.449532 (+ 3717us) server_negotiation.cc:407] Connection header received
0927 21:16:23.449618 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:23.449619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:23.449628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:23.449676 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:23.450291 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:23.450293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:23.450293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:23.450298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:23.450316 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:23.450320 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:23.450381 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:23.450438 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 21:16:23.450453 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:16:23.452857 21206 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:23.453034 21206 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:23.457852 21206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ad03f8e2b154c61aae6fd90e5701228: Generated new TSK 0
I20260927 21:16:23.458058 21206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:23.461377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 21127
2026-09-27T21:16:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:23 dist-test-slave-r9x1 krb5kdc[21226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:23 dist-test-slave-r9x1 krb5kdc[21226](info): set up 2 sockets
Sep 27 21:16:23 dist-test-slave-r9x1 krb5kdc[21226](info): commencing operation
krb5kdc: starting...
W20260927 21:16:24.429162 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:24 dist-test-slave-r9x1 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:24.552069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:24.636525 21239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:24.636762 21239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:24.636794 21239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:24.638458 21239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:24.638542 21239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:24.638557 21239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:24.638568 21239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:24.638579 21239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:24.640352 21239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:24.640637 21239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:24.641026 21239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:24.646031 21247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:24.646031 21244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:24.646044 21245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:24.646414 21239 server_base.cc:1034] running on GCE node
I20260927 21:16:24.646646 21239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:24.646906 21239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:24.648069 21239 hybrid_clock.cc:648] HybridClock initialized: now 1790543784648051 us; error 32 us; skew 500 ppm
Sep 27 21:16:24 dist-test-slave-r9x1 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:24.650772 21239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:24.651441 21239 webserver.cc:460] Webserver started at http://127.0.0.1:32959/ using document root <none> and password file <none>
I20260927 21:16:24.651693 21239 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:24.651746 21239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:24.651880 21239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:24.652961 21239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "64a1fd115ab240a49f26f408bcab752f"
format_stamp: "Formatted at 2026-09-27 21:16:24 on dist-test-slave-r9x1"
server_key: "1532f55ef347853063b16c33cd8d1b33"
server_key_iv: "80fc6281865448f0774ae61d3cab202f"
server_key_version: "encryptionkey@0"
I20260927 21:16:24.653354 21239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "64a1fd115ab240a49f26f408bcab752f"
format_stamp: "Formatted at 2026-09-27 21:16:24 on dist-test-slave-r9x1"
server_key: "1532f55ef347853063b16c33cd8d1b33"
server_key_iv: "80fc6281865448f0774ae61d3cab202f"
server_key_version: "encryptionkey@0"
I20260927 21:16:24.654682 21239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:24.655398 21254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:24.655550 21239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:16:24.655642 21239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "64a1fd115ab240a49f26f408bcab752f"
format_stamp: "Formatted at 2026-09-27 21:16:24 on dist-test-slave-r9x1"
server_key: "1532f55ef347853063b16c33cd8d1b33"
server_key_iv: "80fc6281865448f0774ae61d3cab202f"
server_key_version: "encryptionkey@0"
I20260927 21:16:24.655707 21239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:24.678360 21239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:24.680430 21239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:24.680572 21239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:24.684598 21239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44309
I20260927 21:16:24.684638 21306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44309 every 8 connection(s)
I20260927 21:16:24.685002 21239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:24.685871 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:24.687367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 21239
I20260927 21:16:24.687486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:24.687681 392 external_mini_cluster.cc:1428] Setting key 3f18df74d96daf1a499b4619e7a73119
I20260927 21:16:24.688527 21307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f: Bootstrap starting.
I20260927 21:16:24.689118 21307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:24.689435 21307 log.cc:824] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:24.690089 21307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f: No bootstrap required, opened a new log
I20260927 21:16:24.691910 21307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1fd115ab240a49f26f408bcab752f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } }
I20260927 21:16:24.692065 21307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:24.692095 21307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a1fd115ab240a49f26f408bcab752f, State: Initialized, Role: FOLLOWER
I20260927 21:16:24.692226 21307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1fd115ab240a49f26f408bcab752f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } }
I20260927 21:16:24.692323 21307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:24.692355 21307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:24.692396 21307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:24.693017 21307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1fd115ab240a49f26f408bcab752f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } }
I20260927 21:16:24.693113 21307 leader_election.cc:302] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a1fd115ab240a49f26f408bcab752f; no voters:
I20260927 21:16:24.693305 21307 leader_election.cc:288] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:24.693420 21312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:24.693629 21312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [term 1 LEADER]: Becoming Leader. State: Replica: 64a1fd115ab240a49f26f408bcab752f, State: Running, Role: LEADER
I20260927 21:16:24.693698 21307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:24.693725 21312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1fd115ab240a49f26f408bcab752f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } }
I20260927 21:16:24.694069 21312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a1fd115ab240a49f26f408bcab752f. Latest consensus state: current_term: 1 leader_uuid: "64a1fd115ab240a49f26f408bcab752f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1fd115ab240a49f26f408bcab752f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } } }
I20260927 21:16:24.694142 21312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:24.694314 21313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a1fd115ab240a49f26f408bcab752f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a1fd115ab240a49f26f408bcab752f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44309 } } }
I20260927 21:16:24.694377 21313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f [sys.catalog]: This master's current role is: LEADER
Sep 27 21:16:24 dist-test-slave-r9x1 krb5kdc[21226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260927 21:16:24.695380 21328 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:16:24.695426 21328 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:24.695745 21319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:24.695919 21319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:24.697423 21319 catalog_manager.cc:1347] Generated new cluster ID: f95ff7cc22194d90a1f9f44caf629b16
I20260927 21:16:24.697484 21319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:24.699016 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:24.688239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:53143 (local address 127.0.0.1:44309)
0927 21:16:24.688426 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:24.688434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:24.688460 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:16:24.688936 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:24.688943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:24.688995 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:24.689124 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:24.689835 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:24.690779 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:24.691506 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:24.691926 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:24.692020 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:24.695182 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:24.695187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:24.695187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:24.695202 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:24.697092 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:24.697632 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:24.697636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:24.697636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:24.697730 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:24.697947 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:24.697948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:24.697948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:24.698190 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:24.698334 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:24.698483 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 21:16:24.698587 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260927 21:16:24.700023 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:24.707576 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:24.701539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:60395 (local address 127.0.0.1:44309)
0927 21:16:24.701578 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:24.701580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:24.701747 (+ 167us) server_negotiation.cc:407] Connection header received
0927 21:16:24.701806 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:24.701807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:24.701817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:24.701857 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:24.702376 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:24.702897 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:24.703569 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:24.703796 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:24.703853 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:24.705123 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:24.705124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:24.705125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:24.705133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:24.706472 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:24.706839 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:24.706840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:24.706840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:24.706896 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:24.707083 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:24.707083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:24.707084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:24.707199 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:24.707301 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:24.707418 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:16:24.707482 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:16:24.718772 21319 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:24.719514 21319 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:24.725812 21319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a1fd115ab240a49f26f408bcab752f: Generated new TSK 0
I20260927 21:16:24.726087 21319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:24.728796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 21239
W20260927 21:16:24.732707 21224 connection.cc:535] client connection to 127.0.0.1:44309 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44309): Connection reset by peer (error 104)
2026-09-27T21:16:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1266 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:24 dist-test-slave-r9x1 krb5kdc[21338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:24 dist-test-slave-r9x1 krb5kdc[21338](info): set up 2 sockets
Sep 27 21:16:24 dist-test-slave-r9x1 krb5kdc[21338](info): commencing operation
krb5kdc: starting...
W20260927 21:16:25.693799 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:25 dist-test-slave-r9x1 krb5kdc[21338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:25.821916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:25.905676 21351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:25.905921 21351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:25.905957 21351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:25.907284 21351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:25.907337 21351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:25.907356 21351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:25.907372 21351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:25.907387 21351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:25.908987 21351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:25.909255 21351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:25.909507 21351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:25.914527 21359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:25.914520 21351 server_base.cc:1034] running on GCE node
W20260927 21:16:25.914520 21357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:25.914518 21356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:25.914925 21351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:25.915234 21351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:25.916396 21351 hybrid_clock.cc:648] HybridClock initialized: now 1790543785916370 us; error 39 us; skew 500 ppm
Sep 27 21:16:25 dist-test-slave-r9x1 krb5kdc[21338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:25.919008 21351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:25.919482 21351 webserver.cc:460] Webserver started at http://127.0.0.1:40613/ using document root <none> and password file <none>
I20260927 21:16:25.919704 21351 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:25.919749 21351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:25.919865 21351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:25.920938 21351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "c6b29911032740b582fd4179c67334b8"
format_stamp: "Formatted at 2026-09-27 21:16:25 on dist-test-slave-r9x1"
server_key: "5e3956997e39b580f973d6aa3f846715"
server_key_iv: "5ae94761b80914f0b14b7ef0ec9f6303"
server_key_version: "encryptionkey@0"
I20260927 21:16:25.921370 21351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6b29911032740b582fd4179c67334b8"
format_stamp: "Formatted at 2026-09-27 21:16:25 on dist-test-slave-r9x1"
server_key: "5e3956997e39b580f973d6aa3f846715"
server_key_iv: "5ae94761b80914f0b14b7ef0ec9f6303"
server_key_version: "encryptionkey@0"
I20260927 21:16:25.922737 21351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:25.923540 21366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:25.923751 21351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:25.923837 21351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "c6b29911032740b582fd4179c67334b8"
format_stamp: "Formatted at 2026-09-27 21:16:25 on dist-test-slave-r9x1"
server_key: "5e3956997e39b580f973d6aa3f846715"
server_key_iv: "5ae94761b80914f0b14b7ef0ec9f6303"
server_key_version: "encryptionkey@0"
I20260927 21:16:25.923894 21351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:25.953635 21351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:25.956622 21351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:25.956780 21351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:25.960914 21351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35133
I20260927 21:16:25.960949 21418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35133 every 8 connection(s)
I20260927 21:16:25.961329 21351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:25.962204 21419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:25.964669 21419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8: Bootstrap starting.
I20260927 21:16:25.965291 21419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:25.965567 21419 log.cc:824] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:25.966372 21419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8: No bootstrap required, opened a new log
I20260927 21:16:25.967172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 21351
I20260927 21:16:25.967286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:25.967468 392 external_mini_cluster.cc:1428] Setting key 74137cb354139faad359fc8015ae4d3f
I20260927 21:16:25.968508 21419 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b29911032740b582fd4179c67334b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35133 } }
I20260927 21:16:25.968677 21419 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:25.968725 21419 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b29911032740b582fd4179c67334b8, State: Initialized, Role: FOLLOWER
I20260927 21:16:25.968861 21419 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b29911032740b582fd4179c67334b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35133 } }
I20260927 21:16:25.968945 21419 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:25.968976 21419 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:25.969025 21419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:25.969666 21419 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b29911032740b582fd4179c67334b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35133 } }
I20260927 21:16:25.969767 21419 leader_election.cc:302] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b29911032740b582fd4179c67334b8; no voters:
I20260927 21:16:25.969939 21419 leader_election.cc:288] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:25.970016 21424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:25.970141 21424 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [term 1 LEADER]: Becoming Leader. State: Replica: c6b29911032740b582fd4179c67334b8, State: Running, Role: LEADER
I20260927 21:16:25.970224 21424 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b29911032740b582fd4179c67334b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35133 } }
I20260927 21:16:25.970227 21419 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:25.970546 21424 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b29911032740b582fd4179c67334b8. Latest consensus state: current_term: 1 leader_uuid: "c6b29911032740b582fd4179c67334b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b29911032740b582fd4179c67334b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35133 } } }
I20260927 21:16:25.970556 21425 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b29911032740b582fd4179c67334b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b29911032740b582fd4179c67334b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35133 } } }
I20260927 21:16:25.970628 21424 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:25.970631 21425 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:25.971321 21429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:25.971792 21429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:25.973393 21429 catalog_manager.cc:1347] Generated new cluster ID: 5add18e46801415d9fd3496b6fd6eeb1
I20260927 21:16:25.973448 21429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:25.974280 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:25.967983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:41637 (local address 127.0.0.1:35133)
0927 21:16:25.968161 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:25.968169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:25.968196 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:16:25.968471 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:25.968476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:25.968526 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:25.968640 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:25.968929 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:25.970559 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:25.971743 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:25.972398 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:25.972536 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:25.973342 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:25.973345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:25.973346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:25.973570 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:25.973610 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:25.973627 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:25.973766 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:25.974033 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 21:16:25.974063 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260927 21:16:25.975371 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:25.981298 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:25.977566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:50203 (local address 127.0.0.1:35133)
0927 21:16:25.977610 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:25.977613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:25.977842 (+ 229us) server_negotiation.cc:407] Connection header received
0927 21:16:25.977892 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:25.977893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:25.977917 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:25.977983 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:25.978111 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:25.978891 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:25.979883 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:25.980214 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:25.980286 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:25.981021 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:25.981023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:25.981023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:25.981029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:25.981046 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:25.981051 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:25.981109 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:25.981170 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:16:25.981185 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:16:25.987597 21429 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:25.988360 21429 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:25.996115 21429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6b29911032740b582fd4179c67334b8: Generated new TSK 0
I20260927 21:16:25.996343 21429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:26.004364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 21351
W20260927 21:16:26.008067 21336 connection.cc:535] client connection to 127.0.0.1:35133 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35133): Connection reset by peer (error 104)
2026-09-27T21:16:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:26 dist-test-slave-r9x1 krb5kdc[21450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:26 dist-test-slave-r9x1 krb5kdc[21450](info): set up 2 sockets
Sep 27 21:16:26 dist-test-slave-r9x1 krb5kdc[21450](info): commencing operation
krb5kdc: starting...
W20260927 21:16:26.969576 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:27 dist-test-slave-r9x1 krb5kdc[21450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:27.089555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:27.173012 21463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:27.173243 21463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:27.173316 21463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:27.174583 21463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:27.174629 21463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:27.174642 21463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:27.174654 21463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:27.174665 21463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:27.176190 21463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:27.176405 21463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:27.176594 21463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:27.181834 21469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:27.181834 21471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:27.181854 21468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:27.181856 21463 server_base.cc:1034] running on GCE node
I20260927 21:16:27.182327 21463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:27.182611 21463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:27.183763 21463 hybrid_clock.cc:648] HybridClock initialized: now 1790543787183740 us; error 37 us; skew 500 ppm
Sep 27 21:16:27 dist-test-slave-r9x1 krb5kdc[21450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:27.186219 21463 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:27.186678 21463 webserver.cc:460] Webserver started at http://127.0.0.1:33537/ using document root <none> and password file <none>
I20260927 21:16:27.186898 21463 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:27.186946 21463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:27.187071 21463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:27.188128 21463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "26a63c54af8240f0ae0874aa41f11b4c"
format_stamp: "Formatted at 2026-09-27 21:16:27 on dist-test-slave-r9x1"
server_key: "2a9995ca7ffec796b6eb3b6cb44b27af"
server_key_iv: "f438a6397452b621cded4d0e2fcb8423"
server_key_version: "encryptionkey@0"
I20260927 21:16:27.188469 21463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "26a63c54af8240f0ae0874aa41f11b4c"
format_stamp: "Formatted at 2026-09-27 21:16:27 on dist-test-slave-r9x1"
server_key: "2a9995ca7ffec796b6eb3b6cb44b27af"
server_key_iv: "f438a6397452b621cded4d0e2fcb8423"
server_key_version: "encryptionkey@0"
I20260927 21:16:27.190007 21463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:27.190835 21478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:27.191069 21463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:27.191156 21463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "26a63c54af8240f0ae0874aa41f11b4c"
format_stamp: "Formatted at 2026-09-27 21:16:27 on dist-test-slave-r9x1"
server_key: "2a9995ca7ffec796b6eb3b6cb44b27af"
server_key_iv: "f438a6397452b621cded4d0e2fcb8423"
server_key_version: "encryptionkey@0"
I20260927 21:16:27.191229 21463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:27.194938 21463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:27.197410 21463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:27.197561 21463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:27.201658 21463 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37083
I20260927 21:16:27.201681 21530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37083 every 8 connection(s)
I20260927 21:16:27.202080 21463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:27.203022 21531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:27.204555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 21463
I20260927 21:16:27.204659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:27.204830 392 external_mini_cluster.cc:1428] Setting key 00b3bfe055d4edbc9cc111469e610d85
I20260927 21:16:27.205636 21531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c: Bootstrap starting.
I20260927 21:16:27.206326 21531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:27.206646 21531 log.cc:824] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:27.207382 21531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c: No bootstrap required, opened a new log
I20260927 21:16:27.207386 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:27.205391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:48143 (local address 127.0.0.1:37083)
0927 21:16:27.205591 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:27.205601 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:27.205632 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:16:27.206193 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:27.206203 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:27.206238 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:27.206387 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:27.206955 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:27.206960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:27.206960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:27.206964 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:27.207008 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:27.207023 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:27.207095 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:27.207148 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 21:16:27.207185 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":100,"threads_started":1}
I20260927 21:16:27.208477 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:27.209846 21531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a63c54af8240f0ae0874aa41f11b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260927 21:16:27.210022 21531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:27.210057 21531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a63c54af8240f0ae0874aa41f11b4c, State: Initialized, Role: FOLLOWER
I20260927 21:16:27.210181 21531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a63c54af8240f0ae0874aa41f11b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260927 21:16:27.210286 21531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:27.210319 21531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:27.210355 21531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:27.210952 21531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a63c54af8240f0ae0874aa41f11b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260927 21:16:27.211059 21531 leader_election.cc:302] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26a63c54af8240f0ae0874aa41f11b4c; no voters:
I20260927 21:16:27.211241 21531 leader_election.cc:288] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:27.211342 21541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:27.211498 21531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:27.211498 21541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [term 1 LEADER]: Becoming Leader. State: Replica: 26a63c54af8240f0ae0874aa41f11b4c, State: Running, Role: LEADER
I20260927 21:16:27.211628 21541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a63c54af8240f0ae0874aa41f11b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260927 21:16:27.211719 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:27.210246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:46127 (local address 127.0.0.1:37083)
0927 21:16:27.210329 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:27.210332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:27.210680 (+ 348us) server_negotiation.cc:407] Connection header received
0927 21:16:27.210771 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:27.210772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:27.210778 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:27.210825 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:27.211436 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:27.211437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:27.211437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:27.211442 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:27.211458 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:27.211463 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:27.211493 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:27.211645 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:16:27.211659 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:16:27.212199 21542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26a63c54af8240f0ae0874aa41f11b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a63c54af8240f0ae0874aa41f11b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } } }
I20260927 21:16:27.212212 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26a63c54af8240f0ae0874aa41f11b4c. Latest consensus state: current_term: 1 leader_uuid: "26a63c54af8240f0ae0874aa41f11b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a63c54af8240f0ae0874aa41f11b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } } }
I20260927 21:16:27.212368 21542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:27.212385 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c [sys.catalog]: This master's current role is: LEADER
W20260927 21:16:27.213297 21555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:16:27.213353 21555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:27.213393 21557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:27.213560 21557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:27.214995 21557 catalog_manager.cc:1347] Generated new cluster ID: 65df643de7df408aa30879c367e88281
I20260927 21:16:27.215054 21557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:27.233971 21557 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:27.234167 21557 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:27.238229 21557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26a63c54af8240f0ae0874aa41f11b4c: Generated new TSK 0
I20260927 21:16:27.238435 21557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:27.244730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 21463
2026-09-27T21:16:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1240 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:27 dist-test-slave-r9x1 krb5kdc[21562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:27 dist-test-slave-r9x1 krb5kdc[21562](info): set up 2 sockets
Sep 27 21:16:27 dist-test-slave-r9x1 krb5kdc[21562](info): commencing operation
krb5kdc: starting...
W20260927 21:16:28.201392 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:28 dist-test-slave-r9x1 krb5kdc[21562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:28.320603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:28.404518 21575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:28.404760 21575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:28.404786 21575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:28.406276 21575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:28.406338 21575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:28.406351 21575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:28.406363 21575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:28.406374 21575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:28.407940 21575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35567
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:28.408170 21575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:28.408354 21575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:28.413061 21581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:28.413041 21583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:28.413046 21580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:28.413386 21575 server_base.cc:1034] running on GCE node
I20260927 21:16:28.413612 21575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:28.413895 21575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:28.415051 21575 hybrid_clock.cc:648] HybridClock initialized: now 1790543788415035 us; error 32 us; skew 500 ppm
Sep 27 21:16:28 dist-test-slave-r9x1 krb5kdc[21562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:28.421094 21575 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:28.421597 21575 webserver.cc:460] Webserver started at http://127.0.0.1:45373/ using document root <none> and password file <none>
I20260927 21:16:28.421799 21575 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:28.421854 21575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:28.421981 21575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:28.423067 21575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "d859d5cc1df24f7dac7510d2733c5982"
format_stamp: "Formatted at 2026-09-27 21:16:28 on dist-test-slave-r9x1"
server_key: "a231629fa24af6d962f3de7c0f9a6ec2"
server_key_iv: "befb0bcadb6d47bdded7b74d8c0d160e"
server_key_version: "encryptionkey@0"
I20260927 21:16:28.423400 21575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "d859d5cc1df24f7dac7510d2733c5982"
format_stamp: "Formatted at 2026-09-27 21:16:28 on dist-test-slave-r9x1"
server_key: "a231629fa24af6d962f3de7c0f9a6ec2"
server_key_iv: "befb0bcadb6d47bdded7b74d8c0d160e"
server_key_version: "encryptionkey@0"
I20260927 21:16:28.424759 21575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:16:28.425573 21590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:28.425712 21575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:16:28.425796 21575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "d859d5cc1df24f7dac7510d2733c5982"
format_stamp: "Formatted at 2026-09-27 21:16:28 on dist-test-slave-r9x1"
server_key: "a231629fa24af6d962f3de7c0f9a6ec2"
server_key_iv: "befb0bcadb6d47bdded7b74d8c0d160e"
server_key_version: "encryptionkey@0"
I20260927 21:16:28.425872 21575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:28.434304 21575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:28.436178 21575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:28.436312 21575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:28.440048 21575 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35567
I20260927 21:16:28.440115 21642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35567 every 8 connection(s)
I20260927 21:16:28.440438 21575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:28.441309 21643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:28.443555 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982: Bootstrap starting.
I20260927 21:16:28.444088 21643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:28.444314 21643 log.cc:824] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:28.444947 21643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982: No bootstrap required, opened a new log
I20260927 21:16:28.445396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 21575
I20260927 21:16:28.445523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:28.445711 392 external_mini_cluster.cc:1428] Setting key 881b48b58860dcf348d9f45625b044e8
I20260927 21:16:28.446957 21643 raft_consensus.cc:348] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d859d5cc1df24f7dac7510d2733c5982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } }
I20260927 21:16:28.447131 21643 raft_consensus.cc:374] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:28.447167 21643 raft_consensus.cc:729] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d859d5cc1df24f7dac7510d2733c5982, State: Initialized, Role: FOLLOWER
I20260927 21:16:28.447312 21643 consensus_queue.cc:260] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d859d5cc1df24f7dac7510d2733c5982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } }
I20260927 21:16:28.447387 21643 raft_consensus.cc:388] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:28.447408 21643 raft_consensus.cc:482] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:28.447454 21643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:28.448513 21643 raft_consensus.cc:504] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d859d5cc1df24f7dac7510d2733c5982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } }
I20260927 21:16:28.448625 21643 leader_election.cc:302] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d859d5cc1df24f7dac7510d2733c5982; no voters:
I20260927 21:16:28.448791 21643 leader_election.cc:288] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:28.448836 21648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:28.448998 21648 raft_consensus.cc:686] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [term 1 LEADER]: Becoming Leader. State: Replica: d859d5cc1df24f7dac7510d2733c5982, State: Running, Role: LEADER
I20260927 21:16:28.449033 21643 sys_catalog.cc:564] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:28.449088 21648 consensus_queue.cc:237] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d859d5cc1df24f7dac7510d2733c5982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } }
I20260927 21:16:28.449482 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d859d5cc1df24f7dac7510d2733c5982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d859d5cc1df24f7dac7510d2733c5982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } } }
I20260927 21:16:28.449494 21650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d859d5cc1df24f7dac7510d2733c5982. Latest consensus state: current_term: 1 leader_uuid: "d859d5cc1df24f7dac7510d2733c5982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d859d5cc1df24f7dac7510d2733c5982" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } } }
I20260927 21:16:28.449579 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:28.449592 21650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:28.449765 21657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:28.450239 21657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:16:28 dist-test-slave-r9x1 krb5kdc[21562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:16:28.454767 21657 catalog_manager.cc:1347] Generated new cluster ID: 3ed3a96ff1624f21b803eadd0d6fb27f
I20260927 21:16:28.454849 21657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:28.457172 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:28.446276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:33311 (local address 127.0.0.1:35567)
0927 21:16:28.446468 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:28.446478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:28.446516 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:16:28.446922 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:28.446930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:28.447005 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:28.447151 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:28.447842 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:28.449193 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:28.450320 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:28.450901 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:28.451021 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:28.453696 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:28.453701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:28.453701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:28.453715 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:28.455421 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:28.456001 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:28.456004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:28.456004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:28.456125 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:28.456346 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:28.456346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:28.456347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:28.456589 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:28.456758 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:28.456881 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 21:16:28.456986 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260927 21:16:28.458179 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:28.471246 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:28.461586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:44835 (local address 127.0.0.1:35567)
0927 21:16:28.461628 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:28.461630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:28.465479 (+ 3849us) server_negotiation.cc:407] Connection header received
0927 21:16:28.465549 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:28.465551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:28.465565 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:28.465633 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:28.466231 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:28.466737 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:28.467422 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:28.467697 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:28.467756 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:28.468912 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:28.468914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:28.468914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:28.468922 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:28.470197 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:28.470533 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:28.470534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:28.470534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:28.470586 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:28.470775 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:28.470776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:28.470776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:28.470884 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:28.470998 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:28.471090 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:16:28.471151 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:16:28.493376 21657 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:28.494133 21657 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:28.499586 21657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d859d5cc1df24f7dac7510d2733c5982: Generated new TSK 0
I20260927 21:16:28.499832 21657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:28.508869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 21575
W20260927 21:16:28.513099 21560 connection.cc:535] client connection to 127.0.0.1:35567 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35567): Connection reset by peer (error 104)
2026-09-27T21:16:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:28 dist-test-slave-r9x1 krb5kdc[21674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:28 dist-test-slave-r9x1 krb5kdc[21674](info): set up 2 sockets
Sep 27 21:16:28 dist-test-slave-r9x1 krb5kdc[21674](info): commencing operation
krb5kdc: starting...
W20260927 21:16:29.473431 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:29 dist-test-slave-r9x1 krb5kdc[21674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:29.596056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:29.683082 21687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:29.683313 21687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:29.683346 21687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:29.684697 21687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:29.684746 21687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:29.684765 21687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:29.684780 21687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:29.684798 21687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:29.686460 21687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:29.686748 21687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:29.686973 21687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:29.691838 21695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:29.691803 21692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:29.691812 21693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:29.692047 21687 server_base.cc:1034] running on GCE node
I20260927 21:16:29.692557 21687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:29.692979 21687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:29.694262 21687 hybrid_clock.cc:648] HybridClock initialized: now 1790543789694244 us; error 30 us; skew 500 ppm
Sep 27 21:16:29 dist-test-slave-r9x1 krb5kdc[21674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:29.696898 21687 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:29.697438 21687 webserver.cc:460] Webserver started at http://127.0.0.1:43959/ using document root <none> and password file <none>
I20260927 21:16:29.697685 21687 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:29.697734 21687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:29.697862 21687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:29.698961 21687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "0facd66988134ace99ba3faada1cefc3"
format_stamp: "Formatted at 2026-09-27 21:16:29 on dist-test-slave-r9x1"
server_key: "285696ca0f2b40557738b4798d793f39"
server_key_iv: "42d531be4d09a9f17b91d5a93a61b415"
server_key_version: "encryptionkey@0"
I20260927 21:16:29.699306 21687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "0facd66988134ace99ba3faada1cefc3"
format_stamp: "Formatted at 2026-09-27 21:16:29 on dist-test-slave-r9x1"
server_key: "285696ca0f2b40557738b4798d793f39"
server_key_iv: "42d531be4d09a9f17b91d5a93a61b415"
server_key_version: "encryptionkey@0"
I20260927 21:16:29.700732 21687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:16:29.701588 21702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:29.701805 21687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:29.701903 21687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "0facd66988134ace99ba3faada1cefc3"
format_stamp: "Formatted at 2026-09-27 21:16:29 on dist-test-slave-r9x1"
server_key: "285696ca0f2b40557738b4798d793f39"
server_key_iv: "42d531be4d09a9f17b91d5a93a61b415"
server_key_version: "encryptionkey@0"
I20260927 21:16:29.701969 21687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:29.726951 21687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:29.729206 21687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:29.729494 21687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:29.733433 21687 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32785
I20260927 21:16:29.733480 21754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32785 every 8 connection(s)
I20260927 21:16:29.733825 21687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:29.734826 21755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:29.737179 21755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3: Bootstrap starting.
I20260927 21:16:29.737947 21755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:29.738226 21755 log.cc:824] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:29.739005 21755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3: No bootstrap required, opened a new log
I20260927 21:16:29.740686 21755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0facd66988134ace99ba3faada1cefc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260927 21:16:29.740852 21755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:29.740875 21755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0facd66988134ace99ba3faada1cefc3, State: Initialized, Role: FOLLOWER
I20260927 21:16:29.741004 21755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0facd66988134ace99ba3faada1cefc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260927 21:16:29.741075 21755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:29.741097 21755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:29.741127 21755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:29.741550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 21687
I20260927 21:16:29.741686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:29.741780 21755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0facd66988134ace99ba3faada1cefc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260927 21:16:29.741886 392 external_mini_cluster.cc:1428] Setting key 027cbce025016a7f5d129e53a7531513
I20260927 21:16:29.741886 21755 leader_election.cc:302] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0facd66988134ace99ba3faada1cefc3; no voters:
I20260927 21:16:29.742093 21755 leader_election.cc:288] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:29.742120 21758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:29.742262 21758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0facd66988134ace99ba3faada1cefc3, State: Running, Role: LEADER
I20260927 21:16:29.742393 21758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0facd66988134ace99ba3faada1cefc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } }
I20260927 21:16:29.742391 21755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:29.742882 21760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0facd66988134ace99ba3faada1cefc3. Latest consensus state: current_term: 1 leader_uuid: "0facd66988134ace99ba3faada1cefc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0facd66988134ace99ba3faada1cefc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } } }
I20260927 21:16:29.742964 21760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:29.743009 21759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0facd66988134ace99ba3faada1cefc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0facd66988134ace99ba3faada1cefc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32785 } } }
I20260927 21:16:29.743080 21759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:29.743798 21765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:29.744336 21765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:29.746083 21765 catalog_manager.cc:1347] Generated new cluster ID: 6edd71e7bf444611ab5450563328615f
I20260927 21:16:29.746146 21765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:29.748356 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:29.742367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:37949 (local address 127.0.0.1:32785)
0927 21:16:29.742530 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:29.742563 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:29.743052 (+ 489us) server_negotiation.cc:407] Connection header received
0927 21:16:29.743184 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:29.743192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:29.743257 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:29.743402 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:29.743681 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:29.744810 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:29.745843 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:29.746497 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:29.746757 (+ 260us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:29.747647 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:29.747653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:29.747653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:29.747852 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:29.747890 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:29.747904 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:29.747978 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:29.748083 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:16:29.748116 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260927 21:16:29.749322 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:29.753949 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:29.751156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:59735 (local address 127.0.0.1:32785)
0927 21:16:29.751236 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:29.751238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:29.751246 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:29.751266 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:29.751266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:29.751276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:29.751345 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:29.751500 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:29.752031 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:29.752729 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:29.752954 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:29.753010 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:29.753707 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:29.753709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:29.753709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:29.753714 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:29.753730 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:29.753735 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:29.753779 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:29.753853 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:16:29.753865 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:16:29.762247 21765 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:29.763025 21765 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:29.766593 21765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0facd66988134ace99ba3faada1cefc3: Generated new TSK 0
I20260927 21:16:29.766800 21765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:29.770174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 21687
W20260927 21:16:29.773906 21672 connection.cc:535] client connection to 127.0.0.1:32785 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32785): Connection reset by peer (error 104)
2026-09-27T21:16:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1260 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:29 dist-test-slave-r9x1 krb5kdc[21786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:29 dist-test-slave-r9x1 krb5kdc[21786](info): set up 2 sockets
Sep 27 21:16:29 dist-test-slave-r9x1 krb5kdc[21786](info): commencing operation
krb5kdc: starting...
W20260927 21:16:30.741247 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:30 dist-test-slave-r9x1 krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:16:30.854429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:30.935638 21799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:30.935861 21799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:30.935885 21799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:30.937091 21799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:30.937139 21799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:30.937151 21799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:30.937165 21799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:30.937186 21799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:30.938812 21799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:30.939069 21799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:30.939301 21799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:30.944051 21805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:30.944067 21804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:30.944041 21807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:30.944235 21799 server_base.cc:1034] running on GCE node
I20260927 21:16:30.944509 21799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:30.944802 21799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:30.945968 21799 hybrid_clock.cc:648] HybridClock initialized: now 1790543790945952 us; error 31 us; skew 500 ppm
Sep 27 21:16:30 dist-test-slave-r9x1 krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:30.948601 21799 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:16:30.949077 21799 webserver.cc:460] Webserver started at http://127.0.0.1:38649/ using document root <none> and password file <none>
I20260927 21:16:30.949425 21799 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:30.949494 21799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:30.949636 21799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:30.950724 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "8a4ed3be017349e89ec88ed1e5821576"
format_stamp: "Formatted at 2026-09-27 21:16:30 on dist-test-slave-r9x1"
server_key: "93d8340f53bbbc83f31f1b0073a26fe3"
server_key_iv: "2d240ee56c9e36c67214d17b44e3a247"
server_key_version: "encryptionkey@0"
I20260927 21:16:30.951062 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a4ed3be017349e89ec88ed1e5821576"
format_stamp: "Formatted at 2026-09-27 21:16:30 on dist-test-slave-r9x1"
server_key: "93d8340f53bbbc83f31f1b0073a26fe3"
server_key_iv: "2d240ee56c9e36c67214d17b44e3a247"
server_key_version: "encryptionkey@0"
I20260927 21:16:30.952596 21799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:30.953568 21814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:30.953781 21799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:30.953895 21799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "8a4ed3be017349e89ec88ed1e5821576"
format_stamp: "Formatted at 2026-09-27 21:16:30 on dist-test-slave-r9x1"
server_key: "93d8340f53bbbc83f31f1b0073a26fe3"
server_key_iv: "2d240ee56c9e36c67214d17b44e3a247"
server_key_version: "encryptionkey@0"
I20260927 21:16:30.953962 21799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:30.957311 21799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:30.959493 21799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:30.959632 21799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:30.963413 21799 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42353
I20260927 21:16:30.963451 21866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42353 every 8 connection(s)
I20260927 21:16:30.963755 21799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:30.964723 21867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:30.967044 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576: Bootstrap starting.
I20260927 21:16:30.967598 21867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:30.967844 21867 log.cc:824] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:30.968461 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576: No bootstrap required, opened a new log
I20260927 21:16:30.969197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 21799
I20260927 21:16:30.969333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:30.969527 392 external_mini_cluster.cc:1428] Setting key b9f21e25799196a9d935312a598845c9
I20260927 21:16:30.970494 21867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ed3be017349e89ec88ed1e5821576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260927 21:16:30.970667 21867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:30.970705 21867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ed3be017349e89ec88ed1e5821576, State: Initialized, Role: FOLLOWER
I20260927 21:16:30.970883 21867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ed3be017349e89ec88ed1e5821576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260927 21:16:30.970971 21867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:30.971004 21867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:30.971052 21867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:30.971767 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:30.970043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:38955 (local address 127.0.0.1:42353)
0927 21:16:30.970218 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:30.970228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:30.970257 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:16:30.970464 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:30.970470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:30.970496 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:30.970611 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:30.971336 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:30.971341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:30.971341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:30.971346 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:30.971397 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:30.971408 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:30.971466 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:30.971533 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 21:16:30.971584 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260927 21:16:30.972074 21867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ed3be017349e89ec88ed1e5821576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260927 21:16:30.972182 21867 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a4ed3be017349e89ec88ed1e5821576; no voters:
I20260927 21:16:30.972350 21867 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:30.972384 21872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:30.972615 21867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:30.972678 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:30.972618 21872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [term 1 LEADER]: Becoming Leader. State: Replica: 8a4ed3be017349e89ec88ed1e5821576, State: Running, Role: LEADER
I20260927 21:16:30.972831 21872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ed3be017349e89ec88ed1e5821576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260927 21:16:30.973203 21872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4ed3be017349e89ec88ed1e5821576. Latest consensus state: current_term: 1 leader_uuid: "8a4ed3be017349e89ec88ed1e5821576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ed3be017349e89ec88ed1e5821576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
I20260927 21:16:30.973313 21872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [sys.catalog]: This master's current role is: LEADER
W20260927 21:16:30.974495 21891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 21:16:30.974764 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a4ed3be017349e89ec88ed1e5821576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ed3be017349e89ec88ed1e5821576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
W20260927 21:16:30.976191 21891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:30.976203 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:30.976292 21881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:30.976500 21881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:30.976609 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:30.974319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:57119 (local address 127.0.0.1:42353)
0927 21:16:30.974355 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:30.974357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:30.974958 (+ 601us) server_negotiation.cc:407] Connection header received
0927 21:16:30.975052 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:30.975053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:30.975059 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:30.975087 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:30.976284 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:30.976285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:30.976286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:30.976291 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:30.976308 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:30.976313 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:30.976347 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:30.976470 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 21:16:30.976486 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:16:30.978129 21881 catalog_manager.cc:1347] Generated new cluster ID: a07d9f0bf7a540d09f13a26b3333cefc
I20260927 21:16:30.978188 21881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:30.993783 21881 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:30.993989 21881 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:31.003127 21881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a4ed3be017349e89ec88ed1e5821576: Generated new TSK 0
I20260927 21:16:31.003317 21881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:31.006223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 21799
2026-09-27T21:16:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:31 dist-test-slave-r9x1 krb5kdc[21898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:31 dist-test-slave-r9x1 krb5kdc[21898](info): set up 2 sockets
Sep 27 21:16:31 dist-test-slave-r9x1 krb5kdc[21898](info): commencing operation
krb5kdc: starting...
W20260927 21:16:31.973177 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:32 dist-test-slave-r9x1 krb5kdc[21898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:32.104391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:32.187062 21911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:32.187286 21911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:32.187309 21911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:32.188513 21911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:32.188568 21911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:32.188580 21911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:32.188592 21911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:32.188602 21911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:32.190187 21911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:32.190438 21911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:32.190693 21911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:32.195779 21917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:32.195972 21919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:32.195808 21916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:32.196151 21911 server_base.cc:1034] running on GCE node
I20260927 21:16:32.196398 21911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:32.196686 21911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:32.197842 21911 hybrid_clock.cc:648] HybridClock initialized: now 1790543792197828 us; error 33 us; skew 500 ppm
Sep 27 21:16:32 dist-test-slave-r9x1 krb5kdc[21898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:32.200875 21911 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:32.201563 21911 webserver.cc:460] Webserver started at http://127.0.98.62:36075/ using document root <none> and password file <none>
I20260927 21:16:32.201805 21911 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:32.201851 21911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:32.201982 21911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:32.203260 21911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "9e98efc040124ac19df60b47c1cd1ba3"
format_stamp: "Formatted at 2026-09-27 21:16:32 on dist-test-slave-r9x1"
server_key: "939fcd7b7748f0abef6b1e5091c92624"
server_key_iv: "c055072cc88119c4f8311613637ca5b5"
server_key_version: "encryptionkey@0"
I20260927 21:16:32.203606 21911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e98efc040124ac19df60b47c1cd1ba3"
format_stamp: "Formatted at 2026-09-27 21:16:32 on dist-test-slave-r9x1"
server_key: "939fcd7b7748f0abef6b1e5091c92624"
server_key_iv: "c055072cc88119c4f8311613637ca5b5"
server_key_version: "encryptionkey@0"
I20260927 21:16:32.205194 21911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:16:32.206408 21926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:32.206629 21911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:32.206727 21911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "9e98efc040124ac19df60b47c1cd1ba3"
format_stamp: "Formatted at 2026-09-27 21:16:32 on dist-test-slave-r9x1"
server_key: "939fcd7b7748f0abef6b1e5091c92624"
server_key_iv: "c055072cc88119c4f8311613637ca5b5"
server_key_version: "encryptionkey@0"
I20260927 21:16:32.206836 21911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:32.229938 21911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:32.232069 21911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:32.232192 21911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:32.235970 21911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35241
I20260927 21:16:32.236014 21978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35241 every 8 connection(s)
I20260927 21:16:32.236464 21911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:32.237382 21979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:32.239578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 21911
I20260927 21:16:32.239669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:32.239738 21979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3: Bootstrap starting.
I20260927 21:16:32.239836 392 external_mini_cluster.cc:1428] Setting key b9b5e7515d62da81c541347abbe30c0e
I20260927 21:16:32.240438 21979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:32.240747 21979 log.cc:824] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:32.241505 21979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3: No bootstrap required, opened a new log
I20260927 21:16:32.243788 21979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e98efc040124ac19df60b47c1cd1ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35241 } }
I20260927 21:16:32.243947 21979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:32.243968 21979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e98efc040124ac19df60b47c1cd1ba3, State: Initialized, Role: FOLLOWER
I20260927 21:16:32.244078 21979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e98efc040124ac19df60b47c1cd1ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35241 } }
I20260927 21:16:32.244151 21979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:32.244175 21979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:32.244222 21979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:32.244935 21979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e98efc040124ac19df60b47c1cd1ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35241 } }
I20260927 21:16:32.245050 21979 leader_election.cc:302] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e98efc040124ac19df60b47c1cd1ba3; no voters:
I20260927 21:16:32.245225 21979 leader_election.cc:288] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:32.245345 21984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:32.245528 21979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:32.245548 21984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 9e98efc040124ac19df60b47c1cd1ba3, State: Running, Role: LEADER
I20260927 21:16:32.245703 21984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e98efc040124ac19df60b47c1cd1ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35241 } }
I20260927 21:16:32.246027 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e98efc040124ac19df60b47c1cd1ba3. Latest consensus state: current_term: 1 leader_uuid: "9e98efc040124ac19df60b47c1cd1ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e98efc040124ac19df60b47c1cd1ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35241 } } }
I20260927 21:16:32.246117 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:32.246052 21986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e98efc040124ac19df60b47c1cd1ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e98efc040124ac19df60b47c1cd1ba3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35241 } } }
I20260927 21:16:32.246260 21986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:32.246381 21991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:32.246999 21991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:32.248989 21991 catalog_manager.cc:1347] Generated new cluster ID: ad72a7ab0a684c9dbafb4b949db99cf3
I20260927 21:16:32.249053 21991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:16:32 dist-test-slave-r9x1 krb5kdc[21898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:32.256433 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:32.240437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39377 (local address 127.0.98.62:35241)
0927 21:16:32.240647 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:32.240656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:32.240694 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:16:32.241247 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:32.241257 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:32.241348 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:32.241494 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:32.242167 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:32.243625 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:32.244565 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:32.245193 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:32.245336 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:32.251945 (+ 6609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:32.251953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:32.251953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:32.251970 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:32.254605 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:32.255111 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:32.255115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:32.255115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:32.255233 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:32.255447 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:32.255448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:32.255448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:32.255803 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:32.255974 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:32.256092 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:16:32.256201 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
I20260927 21:16:32.257649 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:32.257834 21991 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:32.258494 21991 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:32.268924 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:32.259149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51561 (local address 127.0.98.62:35241)
0927 21:16:32.259226 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:32.259228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:32.261470 (+ 2242us) server_negotiation.cc:407] Connection header received
0927 21:16:32.261532 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:32.261533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:32.261545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:32.261607 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:32.262267 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:32.263071 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:32.264182 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:32.264510 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:32.264566 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:32.266113 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:32.266115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:32.266115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:32.266124 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:32.267707 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:32.268155 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:32.268156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:32.268156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:32.268210 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:32.268412 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:32.268412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:32.268412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:32.268530 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:32.268620 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:32.268761 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 21:16:32.268824 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:16:32.270262 21991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e98efc040124ac19df60b47c1cd1ba3: Generated new TSK 0
I20260927 21:16:32.270430 21991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:32.276737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 21911
W20260927 21:16:32.280467 21896 connection.cc:535] client connection to 127.0.98.62:35241 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35241): Connection reset by peer (error 104)
2026-09-27T21:16:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:32 dist-test-slave-r9x1 krb5kdc[22010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:32 dist-test-slave-r9x1 krb5kdc[22010](info): set up 2 sockets
Sep 27 21:16:32 dist-test-slave-r9x1 krb5kdc[22010](info): commencing operation
krb5kdc: starting...
W20260927 21:16:33.229156 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:33 dist-test-slave-r9x1 krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:33.353104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:33.438288 22023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:33.438510 22023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:33.438535 22023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:33.439864 22023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:33.439924 22023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:33.439937 22023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:33.439949 22023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:33.439960 22023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:33.441601 22023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:33.441864 22023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:33.442077 22023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:33.446815 22029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:33.446789 22031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:33.446794 22028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:33.447058 22023 server_base.cc:1034] running on GCE node
I20260927 21:16:33.447310 22023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:33.447602 22023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:33.448737 22023 hybrid_clock.cc:648] HybridClock initialized: now 1790543793448699 us; error 55 us; skew 500 ppm
Sep 27 21:16:33 dist-test-slave-r9x1 krb5kdc[22010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:33.455827 22023 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:33.456314 22023 webserver.cc:460] Webserver started at http://127.0.98.62:44301/ using document root <none> and password file <none>
I20260927 21:16:33.456537 22023 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:33.456586 22023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:33.456725 22023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:33.457875 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "a63ee9ddd4104a1389b0a14565590e86"
format_stamp: "Formatted at 2026-09-27 21:16:33 on dist-test-slave-r9x1"
server_key: "b73d6b72288403fa09f492cc5bed616d"
server_key_iv: "2b3985cdffc627c8bc31d88495810fea"
server_key_version: "encryptionkey@0"
I20260927 21:16:33.458225 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "a63ee9ddd4104a1389b0a14565590e86"
format_stamp: "Formatted at 2026-09-27 21:16:33 on dist-test-slave-r9x1"
server_key: "b73d6b72288403fa09f492cc5bed616d"
server_key_iv: "2b3985cdffc627c8bc31d88495810fea"
server_key_version: "encryptionkey@0"
I20260927 21:16:33.459641 22023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:16:33.460501 22038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:33.460668 22023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:33.460759 22023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "a63ee9ddd4104a1389b0a14565590e86"
format_stamp: "Formatted at 2026-09-27 21:16:33 on dist-test-slave-r9x1"
server_key: "b73d6b72288403fa09f492cc5bed616d"
server_key_iv: "2b3985cdffc627c8bc31d88495810fea"
server_key_version: "encryptionkey@0"
I20260927 21:16:33.460827 22023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:33.476328 22023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:33.478508 22023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:33.478655 22023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:33.482419 22023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39307
I20260927 21:16:33.482468 22090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39307 every 8 connection(s)
I20260927 21:16:33.482831 22023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:33.483665 22091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:33.486059 22091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86: Bootstrap starting.
I20260927 21:16:33.486639 22091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:33.486894 22091 log.cc:824] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:33.487597 22091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86: No bootstrap required, opened a new log
I20260927 21:16:33.488853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 22023
I20260927 21:16:33.488965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:33.489157 392 external_mini_cluster.cc:1428] Setting key 9d17415802ae29d023deb8e671c74b47
I20260927 21:16:33.489903 22091 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63ee9ddd4104a1389b0a14565590e86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } }
I20260927 21:16:33.490126 22091 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:33.490166 22091 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63ee9ddd4104a1389b0a14565590e86, State: Initialized, Role: FOLLOWER
I20260927 21:16:33.490347 22091 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63ee9ddd4104a1389b0a14565590e86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } }
I20260927 21:16:33.490437 22091 raft_consensus.cc:388] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:33.490476 22091 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:33.490522 22091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:33.491838 22091 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63ee9ddd4104a1389b0a14565590e86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } }
I20260927 21:16:33.491956 22091 leader_election.cc:302] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a63ee9ddd4104a1389b0a14565590e86; no voters:
I20260927 21:16:33.492128 22091 leader_election.cc:288] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:33.492182 22096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:33.492373 22091 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:33.492379 22096 raft_consensus.cc:686] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [term 1 LEADER]: Becoming Leader. State: Replica: a63ee9ddd4104a1389b0a14565590e86, State: Running, Role: LEADER
I20260927 21:16:33.492501 22096 consensus_queue.cc:237] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63ee9ddd4104a1389b0a14565590e86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } }
I20260927 21:16:33.492865 22096 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63ee9ddd4104a1389b0a14565590e86. Latest consensus state: current_term: 1 leader_uuid: "a63ee9ddd4104a1389b0a14565590e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63ee9ddd4104a1389b0a14565590e86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } } }
I20260927 21:16:33.492877 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a63ee9ddd4104a1389b0a14565590e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63ee9ddd4104a1389b0a14565590e86" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } } }
I20260927 21:16:33.493039 22096 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:33.493046 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:33.493336 22105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:33.494037 22105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:33.495841 22105 catalog_manager.cc:1347] Generated new cluster ID: e3bddef6e4084d24a02469bde398d67a
I20260927 21:16:33.495896 22105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:33.495975 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:33.490079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43999 (local address 127.0.98.62:39307)
0927 21:16:33.490295 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:33.490306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:33.490359 (+ 53us) server_negotiation.cc:407] Connection header received
0927 21:16:33.490486 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:33.490495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:33.490556 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:33.490711 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:33.490911 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:33.492327 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:33.493531 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:33.494301 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:33.494442 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:33.495211 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:33.495215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:33.495215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:33.495422 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:33.495462 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:33.495477 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:33.495562 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:33.495703 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 21:16:33.495734 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260927 21:16:33.496873 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:33.501811 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:33.498378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40851 (local address 127.0.98.62:39307)
0927 21:16:33.498414 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:33.498415 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:33.498485 (+ 70us) server_negotiation.cc:407] Connection header received
0927 21:16:33.498556 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:33.498556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:33.498566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:33.498617 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:33.498751 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:33.499481 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:33.500442 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:33.500761 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:33.500843 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:33.501561 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:33.501562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:33.501563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:33.501568 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:33.501584 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:33.501589 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:33.501652 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:33.501713 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:16:33.501728 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:16:33.516215 22105 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:33.516949 22105 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:33.524936 22105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a63ee9ddd4104a1389b0a14565590e86: Generated new TSK 0
I20260927 21:16:33.525167 22105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:33.529093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 22023
W20260927 21:16:33.532970 22008 connection.cc:535] client connection to 127.0.98.62:39307 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39307): Connection reset by peer (error 104)
2026-09-27T21:16:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:33 dist-test-slave-r9x1 krb5kdc[22122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:33 dist-test-slave-r9x1 krb5kdc[22122](info): set up 2 sockets
Sep 27 21:16:33 dist-test-slave-r9x1 krb5kdc[22122](info): commencing operation
krb5kdc: starting...
W20260927 21:16:34.489167 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:34 dist-test-slave-r9x1 krb5kdc[22122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:34.608781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:34.689977 22135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:34.690220 22135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:34.690253 22135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:34.691517 22135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:34.691566 22135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:34.691586 22135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:34.691601 22135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:34.691617 22135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:34.693252 22135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:34.693545 22135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:34.693763 22135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:34.698699 22140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:34.698871 22135 server_base.cc:1034] running on GCE node
W20260927 21:16:34.698696 22141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:34.698691 22143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:34.699189 22135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:34.699496 22135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:34.700662 22135 hybrid_clock.cc:648] HybridClock initialized: now 1790543794700637 us; error 39 us; skew 500 ppm
Sep 27 21:16:34 dist-test-slave-r9x1 krb5kdc[22122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:34.703365 22135 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:34.703809 22135 webserver.cc:460] Webserver started at http://127.0.98.62:35407/ using document root <none> and password file <none>
I20260927 21:16:34.704028 22135 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:34.704087 22135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:34.704213 22135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:34.705324 22135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "4668eb0bff004870a1d638c6fdd86f83"
format_stamp: "Formatted at 2026-09-27 21:16:34 on dist-test-slave-r9x1"
server_key: "21398d4fa52d050acfaa3a04c4b8173e"
server_key_iv: "6b9f5a121d30e9aca9cfd47b843439da"
server_key_version: "encryptionkey@0"
I20260927 21:16:34.705673 22135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "4668eb0bff004870a1d638c6fdd86f83"
format_stamp: "Formatted at 2026-09-27 21:16:34 on dist-test-slave-r9x1"
server_key: "21398d4fa52d050acfaa3a04c4b8173e"
server_key_iv: "6b9f5a121d30e9aca9cfd47b843439da"
server_key_version: "encryptionkey@0"
I20260927 21:16:34.707002 22135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:34.707840 22150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:34.708055 22135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:34.708141 22135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "4668eb0bff004870a1d638c6fdd86f83"
format_stamp: "Formatted at 2026-09-27 21:16:34 on dist-test-slave-r9x1"
server_key: "21398d4fa52d050acfaa3a04c4b8173e"
server_key_iv: "6b9f5a121d30e9aca9cfd47b843439da"
server_key_version: "encryptionkey@0"
I20260927 21:16:34.708231 22135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:34.711717 22135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:34.713935 22135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:34.714077 22135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:34.717796 22135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45635
I20260927 21:16:34.717830 22202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45635 every 8 connection(s)
I20260927 21:16:34.718158 22135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:34.719173 22203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:34.721532 22203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83: Bootstrap starting.
I20260927 21:16:34.722249 22203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:34.722501 22203 log.cc:824] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:34.723242 22203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83: No bootstrap required, opened a new log
I20260927 21:16:34.723518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 22135
I20260927 21:16:34.723636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:34.723815 392 external_mini_cluster.cc:1428] Setting key 0b13a7658f072f20e580102eee923d14
I20260927 21:16:34.725615 22203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4668eb0bff004870a1d638c6fdd86f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } }
I20260927 21:16:34.725792 22203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:34.725832 22203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4668eb0bff004870a1d638c6fdd86f83, State: Initialized, Role: FOLLOWER
I20260927 21:16:34.726011 22203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4668eb0bff004870a1d638c6fdd86f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } }
I20260927 21:16:34.726131 22203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:34.726171 22203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:34.726226 22203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:34.726882 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:34.724382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56149 (local address 127.0.98.62:45635)
0927 21:16:34.724554 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:34.724562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:34.724588 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:16:34.725569 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:34.725577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:34.725611 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:34.725767 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:34.726462 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:34.726466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:34.726466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:34.726469 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:34.726507 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:34.726518 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:34.726574 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:34.726639 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 21:16:34.726707 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260927 21:16:34.727325 22203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4668eb0bff004870a1d638c6fdd86f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } }
I20260927 21:16:34.727439 22203 leader_election.cc:302] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4668eb0bff004870a1d638c6fdd86f83; no voters:
I20260927 21:16:34.727614 22203 leader_election.cc:288] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:34.727665 22208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:34.727808 22208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [term 1 LEADER]: Becoming Leader. State: Replica: 4668eb0bff004870a1d638c6fdd86f83, State: Running, Role: LEADER
I20260927 21:16:34.727875 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:34.727918 22203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:34.727948 22208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4668eb0bff004870a1d638c6fdd86f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } }
I20260927 21:16:34.728255 22208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4668eb0bff004870a1d638c6fdd86f83. Latest consensus state: current_term: 1 leader_uuid: "4668eb0bff004870a1d638c6fdd86f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4668eb0bff004870a1d638c6fdd86f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } } }
I20260927 21:16:34.728317 22208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:34.728453 22209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4668eb0bff004870a1d638c6fdd86f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4668eb0bff004870a1d638c6fdd86f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } } }
I20260927 21:16:34.728502 22209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:34.728603 22214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:34.729429 22214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:34.730526 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:34.729448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44901 (local address 127.0.98.62:45635)
0927 21:16:34.729501 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:34.729503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:34.729616 (+ 113us) server_negotiation.cc:407] Connection header received
0927 21:16:34.729703 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:34.729704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:34.729711 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:34.729777 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:34.730326 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:34.730328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:34.730328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:34.730333 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:34.730348 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:34.730352 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:34.730400 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:34.730449 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 21:16:34.730463 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:16:34.734063 22214 catalog_manager.cc:1347] Generated new cluster ID: 7648669f62af4854b89f153414b97ca9
I20260927 21:16:34.734148 22214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:34.749148 22214 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:34.749367 22214 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:34.753829 22214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4668eb0bff004870a1d638c6fdd86f83: Generated new TSK 0
I20260927 21:16:34.754014 22214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:34.763296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 22135
2026-09-27T21:16:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:34 dist-test-slave-r9x1 krb5kdc[22234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:34 dist-test-slave-r9x1 krb5kdc[22234](info): set up 2 sockets
Sep 27 21:16:34 dist-test-slave-r9x1 krb5kdc[22234](info): commencing operation
krb5kdc: starting...
W20260927 21:16:35.721144 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:35 dist-test-slave-r9x1 krb5kdc[22234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:35.840179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43773
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:35.921540 22247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:35.921782 22247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:35.921814 22247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:35.923090 22247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:35.923151 22247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:35.923168 22247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:35.923185 22247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:35.923200 22247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:35.924778 22247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:35.925035 22247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:35.925249 22247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:35.930011 22255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:35.930166 22247 server_base.cc:1034] running on GCE node
W20260927 21:16:35.930011 22252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:35.930033 22253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:35.930516 22247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:35.930823 22247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:35.931998 22247 hybrid_clock.cc:648] HybridClock initialized: now 1790543795931975 us; error 34 us; skew 500 ppm
Sep 27 21:16:35 dist-test-slave-r9x1 krb5kdc[22234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:35.934782 22247 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:35.935246 22247 webserver.cc:460] Webserver started at http://127.0.98.62:38629/ using document root <none> and password file <none>
I20260927 21:16:35.935428 22247 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:35.935467 22247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:35.935557 22247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:35.936573 22247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "c43df301a45b4933a37e0b54d8a6ce21"
format_stamp: "Formatted at 2026-09-27 21:16:35 on dist-test-slave-r9x1"
server_key: "7036a2fc83c2ba60c321c23e9cc4c337"
server_key_iv: "680cbd16abe32915e6f8aec00be2eb23"
server_key_version: "encryptionkey@0"
I20260927 21:16:35.936868 22247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "c43df301a45b4933a37e0b54d8a6ce21"
format_stamp: "Formatted at 2026-09-27 21:16:35 on dist-test-slave-r9x1"
server_key: "7036a2fc83c2ba60c321c23e9cc4c337"
server_key_iv: "680cbd16abe32915e6f8aec00be2eb23"
server_key_version: "encryptionkey@0"
I20260927 21:16:35.938135 22247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:16:35.938973 22262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:35.939157 22247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:35.939240 22247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "c43df301a45b4933a37e0b54d8a6ce21"
format_stamp: "Formatted at 2026-09-27 21:16:35 on dist-test-slave-r9x1"
server_key: "7036a2fc83c2ba60c321c23e9cc4c337"
server_key_iv: "680cbd16abe32915e6f8aec00be2eb23"
server_key_version: "encryptionkey@0"
I20260927 21:16:35.939301 22247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:35.950103 22247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:35.951970 22247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:35.952093 22247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:35.955751 22247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43773
I20260927 21:16:35.955801 22314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43773 every 8 connection(s)
I20260927 21:16:35.956125 22247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:35.957018 22315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:35.959494 22315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21: Bootstrap starting.
I20260927 21:16:35.960114 22315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:35.960386 22315 log.cc:824] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:35.961210 22315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21: No bootstrap required, opened a new log
I20260927 21:16:35.962963 22315 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43df301a45b4933a37e0b54d8a6ce21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260927 21:16:35.963125 22315 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:35.963158 22315 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43df301a45b4933a37e0b54d8a6ce21, State: Initialized, Role: FOLLOWER
I20260927 21:16:35.963304 22315 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43df301a45b4933a37e0b54d8a6ce21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260927 21:16:35.963385 22315 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:35.963418 22315 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:35.963467 22315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:35.964053 22315 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43df301a45b4933a37e0b54d8a6ce21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260927 21:16:35.964155 22315 leader_election.cc:302] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43df301a45b4933a37e0b54d8a6ce21; no voters:
I20260927 21:16:35.964355 22315 leader_election.cc:288] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:35.964442 22318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:35.964609 22315 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:35.964640 22318 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [term 1 LEADER]: Becoming Leader. State: Replica: c43df301a45b4933a37e0b54d8a6ce21, State: Running, Role: LEADER
I20260927 21:16:35.964761 22318 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43df301a45b4933a37e0b54d8a6ce21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } }
I20260927 21:16:35.965212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 22247
I20260927 21:16:35.965360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:35.965379 22319 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43df301a45b4933a37e0b54d8a6ce21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43df301a45b4933a37e0b54d8a6ce21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } } }
I20260927 21:16:35.965445 22319 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:35.965559 392 external_mini_cluster.cc:1428] Setting key 5a1c88d6a9e8904ae90be814b6eee91d
I20260927 21:16:35.965602 22320 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43df301a45b4933a37e0b54d8a6ce21. Latest consensus state: current_term: 1 leader_uuid: "c43df301a45b4933a37e0b54d8a6ce21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43df301a45b4933a37e0b54d8a6ce21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43773 } } }
I20260927 21:16:35.965651 22320 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:35.966008 22327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:35.966532 22327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:35.969180 22327 catalog_manager.cc:1347] Generated new cluster ID: 679babd60b3744b5a6a89f0c833e89c5
I20260927 21:16:35.969247 22327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:16:35 dist-test-slave-r9x1 krb5kdc[22234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:35.980666 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:35.966209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42425 (local address 127.0.98.62:43773)
0927 21:16:35.966356 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:35.966366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:35.966431 (+ 65us) server_negotiation.cc:407] Connection header received
0927 21:16:35.966555 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:35.966562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:35.966629 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:35.966946 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:35.967684 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:35.973310 (+ 5626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:35.974184 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:35.974700 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:35.974809 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:35.977082 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:35.977088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:35.977088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:35.977102 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:35.978923 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:35.979291 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:35.979295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:35.979295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:35.979389 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:35.979746 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:35.979747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:35.979748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:35.980047 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:35.980180 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:35.980321 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 21:16:35.980470 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260927 21:16:35.981726 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:35.985868 22327 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:35.986619 22327 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:35.993091 22327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43df301a45b4933a37e0b54d8a6ce21: Generated new TSK 0
I20260927 21:16:35.993362 22327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:35.995643 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:35.983086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45357 (local address 127.0.98.62:43773)
0927 21:16:35.983136 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:35.983138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:35.989611 (+ 6473us) server_negotiation.cc:407] Connection header received
0927 21:16:35.989685 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:35.989686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:35.989701 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:35.989749 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:35.990314 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:35.990824 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:35.991551 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:35.991768 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:35.991825 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:35.993061 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:35.993063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:35.993063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:35.993071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:35.994595 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:35.994938 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:35.994939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:35.994940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:35.994991 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:35.995154 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:35.995155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:35.995155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:35.995259 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:35.995344 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:35.995446 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:16:35.995530 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:16:35.997864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 22247
2026-09-27T21:16:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:36 dist-test-slave-r9x1 krb5kdc[22346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:36 dist-test-slave-r9x1 krb5kdc[22346](info): set up 2 sockets
Sep 27 21:16:36 dist-test-slave-r9x1 krb5kdc[22346](info): commencing operation
krb5kdc: starting...
W20260927 21:16:36.969184 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:37 dist-test-slave-r9x1 krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:37.088445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:37.169327 22359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:37.169589 22359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:37.169633 22359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:37.170958 22359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:37.171016 22359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:37.171036 22359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:37.171053 22359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:37.171068 22359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:37.172792 22359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:37.173091 22359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:37.173559 22359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:37.178742 22367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:37.178828 22359 server_base.cc:1034] running on GCE node
W20260927 21:16:37.178750 22365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:37.178742 22364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:37.179219 22359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:37.179538 22359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:37.180716 22359 hybrid_clock.cc:648] HybridClock initialized: now 1790543797180693 us; error 35 us; skew 500 ppm
Sep 27 21:16:37 dist-test-slave-r9x1 krb5kdc[22346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:37.183344 22359 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:37.183813 22359 webserver.cc:460] Webserver started at http://127.0.98.62:34561/ using document root <none> and password file <none>
I20260927 21:16:37.184024 22359 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:37.184074 22359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:37.184222 22359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:37.185335 22359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "b477f3a4c0694bb39a6219493c068a39"
format_stamp: "Formatted at 2026-09-27 21:16:37 on dist-test-slave-r9x1"
server_key: "94c684e61c97fe5ca53779f577f86e0b"
server_key_iv: "2ac9e160b1ae89cfa0685794e28d1982"
server_key_version: "encryptionkey@0"
I20260927 21:16:37.185689 22359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "b477f3a4c0694bb39a6219493c068a39"
format_stamp: "Formatted at 2026-09-27 21:16:37 on dist-test-slave-r9x1"
server_key: "94c684e61c97fe5ca53779f577f86e0b"
server_key_iv: "2ac9e160b1ae89cfa0685794e28d1982"
server_key_version: "encryptionkey@0"
I20260927 21:16:37.187242 22359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:37.188235 22374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:37.188486 22359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:37.188584 22359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "b477f3a4c0694bb39a6219493c068a39"
format_stamp: "Formatted at 2026-09-27 21:16:37 on dist-test-slave-r9x1"
server_key: "94c684e61c97fe5ca53779f577f86e0b"
server_key_iv: "2ac9e160b1ae89cfa0685794e28d1982"
server_key_version: "encryptionkey@0"
I20260927 21:16:37.188663 22359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:37.210964 22359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:37.213135 22359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:37.213315 22359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:37.217108 22359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38689
I20260927 21:16:37.217159 22426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38689 every 8 connection(s)
I20260927 21:16:37.217516 22359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:37.218351 22427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:37.220604 22427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39: Bootstrap starting.
I20260927 21:16:37.221149 22427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:37.221442 22427 log.cc:824] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:37.222173 22427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39: No bootstrap required, opened a new log
I20260927 21:16:37.223749 22427 raft_consensus.cc:348] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477f3a4c0694bb39a6219493c068a39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38689 } }
I20260927 21:16:37.223933 22427 raft_consensus.cc:374] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:37.223968 22427 raft_consensus.cc:729] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b477f3a4c0694bb39a6219493c068a39, State: Initialized, Role: FOLLOWER
I20260927 21:16:37.223963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 22359
I20260927 21:16:37.224064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:37.224131 22427 consensus_queue.cc:260] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477f3a4c0694bb39a6219493c068a39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38689 } }
I20260927 21:16:37.224251 22427 raft_consensus.cc:388] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:37.224262 392 external_mini_cluster.cc:1428] Setting key beecaecc36bdd4768f1d53df5dd24421
I20260927 21:16:37.224294 22427 raft_consensus.cc:482] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:37.224336 22427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:37.225322 22427 raft_consensus.cc:504] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477f3a4c0694bb39a6219493c068a39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38689 } }
I20260927 21:16:37.225433 22427 leader_election.cc:302] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b477f3a4c0694bb39a6219493c068a39; no voters:
I20260927 21:16:37.225590 22427 leader_election.cc:288] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:37.225637 22432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:37.225770 22432 raft_consensus.cc:686] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [term 1 LEADER]: Becoming Leader. State: Replica: b477f3a4c0694bb39a6219493c068a39, State: Running, Role: LEADER
I20260927 21:16:37.225834 22427 sys_catalog.cc:564] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:37.225884 22432 consensus_queue.cc:237] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477f3a4c0694bb39a6219493c068a39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38689 } }
I20260927 21:16:37.226244 22433 sys_catalog.cc:455] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b477f3a4c0694bb39a6219493c068a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477f3a4c0694bb39a6219493c068a39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38689 } } }
I20260927 21:16:37.226313 22433 sys_catalog.cc:458] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:37.226497 22434 sys_catalog.cc:455] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b477f3a4c0694bb39a6219493c068a39. Latest consensus state: current_term: 1 leader_uuid: "b477f3a4c0694bb39a6219493c068a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b477f3a4c0694bb39a6219493c068a39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38689 } } }
I20260927 21:16:37.226590 22434 sys_catalog.cc:458] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:37.226845 22439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:37.227370 22439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:37.229513 22439 catalog_manager.cc:1347] Generated new cluster ID: 75272d2bda8d481694ae68e308db35d6
I20260927 21:16:37.229574 22439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:37.230384 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:37.224740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34841 (local address 127.0.98.62:38689)
0927 21:16:37.224936 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:37.224944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:37.224969 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:16:37.225067 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:37.225072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:37.225113 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:37.225229 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:37.225464 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:37.226843 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:37.227942 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:37.228454 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:37.228748 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:37.229646 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:37.229650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:37.229650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:37.229879 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:37.229918 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:37.229934 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:37.230004 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:37.230107 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 21:16:37.230173 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260927 21:16:37.231381 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:37.236250 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:37.232868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56651 (local address 127.0.98.62:38689)
0927 21:16:37.232903 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:37.232905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:37.232965 (+ 60us) server_negotiation.cc:407] Connection header received
0927 21:16:37.233019 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:37.233020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:37.233041 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:37.233102 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:37.233236 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:37.233980 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:37.234957 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:37.235269 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:37.235338 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:37.236006 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:37.236008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:37.236008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:37.236013 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:37.236029 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:37.236033 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:37.236088 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:37.236139 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 21:16:37.236166 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:16:37.237699 22439 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:37.238200 22439 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:37.241873 22439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b477f3a4c0694bb39a6219493c068a39: Generated new TSK 0
I20260927 21:16:37.242050 22439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:37.246326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 22359
2026-09-27T21:16:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:37 dist-test-slave-r9x1 krb5kdc[22458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:37 dist-test-slave-r9x1 krb5kdc[22458](info): set up 2 sockets
Sep 27 21:16:37 dist-test-slave-r9x1 krb5kdc[22458](info): commencing operation
krb5kdc: starting...
W20260927 21:16:38.201123 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:38 dist-test-slave-r9x1 krb5kdc[22458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:38.323287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41093
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:38.404568 22471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:38.404803 22471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:38.404837 22471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:38.406270 22471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:38.406320 22471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:38.406340 22471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:38.406355 22471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:38.406373 22471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:38.407979 22471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:38.408254 22471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:38.408476 22471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:38.413250 22477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:38.413308 22476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:38.413256 22479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:38.413295 22471 server_base.cc:1034] running on GCE node
I20260927 21:16:38.413715 22471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:38.413997 22471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:38.415138 22471 hybrid_clock.cc:648] HybridClock initialized: now 1790543798415125 us; error 30 us; skew 500 ppm
Sep 27 21:16:38 dist-test-slave-r9x1 krb5kdc[22458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:38.417708 22471 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:38.418183 22471 webserver.cc:460] Webserver started at http://127.0.98.62:43507/ using document root <none> and password file <none>
I20260927 21:16:38.418393 22471 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:38.418440 22471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:38.418566 22471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:38.419632 22471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "674141b7cf6d4555ae590755f6a2bffe"
format_stamp: "Formatted at 2026-09-27 21:16:38 on dist-test-slave-r9x1"
server_key: "84a998d9d34f2d32b54267d401a10584"
server_key_iv: "a8b8d8512d09ec733ccf4deef5f39194"
server_key_version: "encryptionkey@0"
I20260927 21:16:38.419962 22471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "674141b7cf6d4555ae590755f6a2bffe"
format_stamp: "Formatted at 2026-09-27 21:16:38 on dist-test-slave-r9x1"
server_key: "84a998d9d34f2d32b54267d401a10584"
server_key_iv: "a8b8d8512d09ec733ccf4deef5f39194"
server_key_version: "encryptionkey@0"
I20260927 21:16:38.421216 22471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:16:38.421940 22486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:38.422122 22471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:16:38.422204 22471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "674141b7cf6d4555ae590755f6a2bffe"
format_stamp: "Formatted at 2026-09-27 21:16:38 on dist-test-slave-r9x1"
server_key: "84a998d9d34f2d32b54267d401a10584"
server_key_iv: "a8b8d8512d09ec733ccf4deef5f39194"
server_key_version: "encryptionkey@0"
I20260927 21:16:38.422273 22471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:38.425738 22471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:38.427598 22471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:38.427742 22471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:38.431499 22471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41093
I20260927 21:16:38.431535 22538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41093 every 8 connection(s)
I20260927 21:16:38.431861 22471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:38.432796 22539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:38.435266 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe: Bootstrap starting.
I20260927 21:16:38.435843 22539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:38.436100 22539 log.cc:824] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:38.436748 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe: No bootstrap required, opened a new log
I20260927 21:16:38.438436 22539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674141b7cf6d4555ae590755f6a2bffe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 21:16:38.438616 22539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:16:38.438640 22539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674141b7cf6d4555ae590755f6a2bffe, State: Initialized, Role: FOLLOWER
I20260927 21:16:38.438776 22539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674141b7cf6d4555ae590755f6a2bffe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 21:16:38.438776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 22471
I20260927 21:16:38.438845 22539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:16:38.438867 22539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:16:38.438887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:38.438915 22539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:38.439074 392 external_mini_cluster.cc:1428] Setting key ae83b2f3f96507189f684dfe2b8b2fae
I20260927 21:16:38.439627 22539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674141b7cf6d4555ae590755f6a2bffe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 21:16:38.439817 22539 leader_election.cc:302] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 674141b7cf6d4555ae590755f6a2bffe; no voters:
I20260927 21:16:38.439990 22539 leader_election.cc:288] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:16:38.440021 22544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:38.440223 22544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [term 1 LEADER]: Becoming Leader. State: Replica: 674141b7cf6d4555ae590755f6a2bffe, State: Running, Role: LEADER
I20260927 21:16:38.440260 22539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:38.440373 22544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674141b7cf6d4555ae590755f6a2bffe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } }
I20260927 21:16:38.440891 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "674141b7cf6d4555ae590755f6a2bffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674141b7cf6d4555ae590755f6a2bffe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } } }
I20260927 21:16:38.440986 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:38.441220 22546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674141b7cf6d4555ae590755f6a2bffe. Latest consensus state: current_term: 1 leader_uuid: "674141b7cf6d4555ae590755f6a2bffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674141b7cf6d4555ae590755f6a2bffe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41093 } } }
I20260927 21:16:38.441437 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:38.439550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38077 (local address 127.0.98.62:41093)
0927 21:16:38.439796 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:38.439807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:38.439846 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:16:38.439976 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:38.439983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:38.440017 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:38.440435 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:38.440822 (+ 387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:38.440825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:38.440826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:38.440830 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:38.440883 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:38.440900 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:38.441109 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:38.441149 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 21:16:38.441190 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":134,"threads_started":1}
I20260927 21:16:38.441440 22546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:38.441633 22549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:38.442134 22549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:38.442494 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:38.443766 22549 catalog_manager.cc:1347] Generated new cluster ID: d9a6da1e456d4c1abf15efabb78aff4d
I20260927 21:16:38.443805 22549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:38.445046 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:38.443889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49297 (local address 127.0.98.62:41093)
0927 21:16:38.443921 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:38.443923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:38.444039 (+ 116us) server_negotiation.cc:407] Connection header received
0927 21:16:38.444142 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:38.444143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:38.444148 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:38.444176 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:38.444818 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:38.444819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:38.444819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:16:38.444824 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:38.444838 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:16:38.444843 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:38.444871 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:38.444970 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 21:16:38.444982 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:16:38.456995 22549 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:38.457162 22549 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:38.465093 22549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 674141b7cf6d4555ae590755f6a2bffe: Generated new TSK 0
I20260927 21:16:38.465305 22549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:38.471387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 22471
2026-09-27T21:16:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1224 ms)
[----------] 18 tests from AuthTokenIssuingTest (22511 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:38 dist-test-slave-r9x1 krb5kdc[22570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:38 dist-test-slave-r9x1 krb5kdc[22570](info): set up 2 sockets
Sep 27 21:16:38 dist-test-slave-r9x1 krb5kdc[22570](info): commencing operation
krb5kdc: starting...
W20260927 21:16:39.425225 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:39.546466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41405,127.0.98.61:40319,127.0.98.60:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:39.629472 22583 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:16:39.629721 22583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:39.629745 22583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:39.629776 22583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:39.631060 22583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:39.631114 22583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:39.631134 22583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:39.631150 22583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:39.631165 22583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:39.632838 22583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41405,127.0.98.61:40319,127.0.98.60:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:39.633112 22583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:39.633462 22583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:39.638451 22589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:39.638497 22583 server_base.cc:1034] running on GCE node
W20260927 21:16:39.638489 22588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:39.638458 22591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:39.638926 22583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:39.639264 22583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:39.640431 22583 hybrid_clock.cc:648] HybridClock initialized: now 1790543799640402 us; error 48 us; skew 500 ppm
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:39.642993 22583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:39.643491 22583 webserver.cc:460] Webserver started at http://127.0.98.62:35855/ using document root <none> and password file <none>
I20260927 21:16:39.643695 22583 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:39.643744 22583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:39.643870 22583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:39.644984 22583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "973669ddb82b4b16837d283f79f3b048"
format_stamp: "Formatted at 2026-09-27 21:16:39 on dist-test-slave-r9x1"
server_key: "33f5c896c382b243b870720fe695ecde"
server_key_iv: "b768a291f11eb9fda3fca7d7c33d397b"
server_key_version: "encryptionkey@0"
I20260927 21:16:39.645373 22583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "973669ddb82b4b16837d283f79f3b048"
format_stamp: "Formatted at 2026-09-27 21:16:39 on dist-test-slave-r9x1"
server_key: "33f5c896c382b243b870720fe695ecde"
server_key_iv: "b768a291f11eb9fda3fca7d7c33d397b"
server_key_version: "encryptionkey@0"
I20260927 21:16:39.646685 22583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:39.647426 22598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:39.647774 22583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:39.647883 22583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "973669ddb82b4b16837d283f79f3b048"
format_stamp: "Formatted at 2026-09-27 21:16:39 on dist-test-slave-r9x1"
server_key: "33f5c896c382b243b870720fe695ecde"
server_key_iv: "b768a291f11eb9fda3fca7d7c33d397b"
server_key_version: "encryptionkey@0"
I20260927 21:16:39.647957 22583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:39.662160 22583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:39.664508 22583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:39.664705 22583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:39.668798 22583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41405
I20260927 21:16:39.668849 22650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41405 every 8 connection(s)
I20260927 21:16:39.669206 22583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:39.670248 22651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:39.671662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 22583
I20260927 21:16:39.671772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:39.671954 392 external_mini_cluster.cc:1428] Setting key 19dfe2bce9a89869925a5825ccbfc6f4
I20260927 21:16:39.672040 22651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 21:16:39.683135 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.672844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41405 (local address 127.0.0.1:33312)
0927 21:16:39.673054 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.673099 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:39.673688 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:39.674089 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:39.674092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:39.674408 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:39.674638 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.674644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.675722 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.675723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.676886 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.676889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.677522 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.677524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.677614 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.678169 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:39.678180 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:39.679833 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:39.681577 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.681584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.681586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.681811 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:39.682043 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.682044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.682044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.682131 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:39.682548 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:39.682556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:39.682638 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 21:16:39.682736 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:16:39.682974 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":66,"thread_start_us":106,"threads_started":1}
I20260927 21:16:39.683141 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.672894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.0.98.62:41405)
0927 21:16:39.673073 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:39.673083 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:39.673206 (+ 123us) server_negotiation.cc:407] Connection header received
0927 21:16:39.673923 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:39.673926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:39.673952 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:39.674062 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:39.674743 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.675588 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.676994 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.677420 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.677510 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.679925 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:39.679933 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:39.679933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:39.679941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:39.681465 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.681890 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.681893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.681894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:39.681978 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.682183 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.682184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.682184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:39.682440 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:39.682538 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:39.682777 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 21:16:39.682878 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260927 21:16:39.684234 22651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:39.684655 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.684480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40319 (local address 127.0.0.1:49310)
0927 21:16:39.684578 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.684631 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 21:16:39.684804 22602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
I20260927 21:16:39.685220 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.685128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40319 (local address 127.0.0.1:49314)
0927 21:16:39.685170 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.685195 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 21:16:39.685322 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40319: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 21:16:39.700150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40319
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41405,127.0.98.61:40319,127.0.98.60:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:16:39.734568 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } attempt: 1
I20260927 21:16:39.735179 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.735000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40319 (local address 127.0.0.1:49326)
0927 21:16:39.735097 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.735146 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:16:39.735555 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.735460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40319 (local address 127.0.0.1:49332)
0927 21:16:39.735508 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.735534 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 21:16:39.735692 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40319: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
W20260927 21:16:39.780279 22658 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:16:39.780526 22658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:39.780551 22658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:39.780578 22658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:39.781810 22658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:39.781860 22658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:39.781878 22658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:39.781895 22658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:39.781910 22658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:39.783499 22658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41405,127.0.98.61:40319,127.0.98.60:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:39.783761 22658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:39.783973 22658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:39.788986 22663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:39.789040 22666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:39.789011 22658 server_base.cc:1034] running on GCE node
W20260927 21:16:39.789002 22664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:39.789464 22658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:39.789768 22658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:39.790930 22658 hybrid_clock.cc:648] HybridClock initialized: now 1790543799790911 us; error 37 us; skew 500 ppm
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:39.793552 22658 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 21:16:39.794018 22658 webserver.cc:460] Webserver started at http://127.0.98.61:42305/ using document root <none> and password file <none>
I20260927 21:16:39.794253 22658 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:39.794305 22658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:39.794435 22658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:39.795524 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/data/instance:
uuid: "e1fa9783664d44be99a935283f5b75c4"
format_stamp: "Formatted at 2026-09-27 21:16:39 on dist-test-slave-r9x1"
server_key: "a32e000cbd1deca820fb9d3fee0e5391"
server_key_iv: "eddf592014c601cd5a277d87c2bf2986"
server_key_version: "encryptionkey@0"
I20260927 21:16:39.795867 22658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/wal/instance:
uuid: "e1fa9783664d44be99a935283f5b75c4"
format_stamp: "Formatted at 2026-09-27 21:16:39 on dist-test-slave-r9x1"
server_key: "a32e000cbd1deca820fb9d3fee0e5391"
server_key_iv: "eddf592014c601cd5a277d87c2bf2986"
server_key_version: "encryptionkey@0"
I20260927 21:16:39.797178 22658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:16:39.798039 22673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:39.798242 22658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:39.798331 22658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/wal
uuid: "e1fa9783664d44be99a935283f5b75c4"
format_stamp: "Formatted at 2026-09-27 21:16:39 on dist-test-slave-r9x1"
server_key: "a32e000cbd1deca820fb9d3fee0e5391"
server_key_iv: "eddf592014c601cd5a277d87c2bf2986"
server_key_version: "encryptionkey@0"
I20260927 21:16:39.798398 22658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:39.803853 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } attempt: 2
I20260927 21:16:39.804344 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.804192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40319 (local address 127.0.0.1:49346)
0927 21:16:39.804258 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.804305 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 21:16:39.804803 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.804707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40319 (local address 127.0.0.1:49354)
0927 21:16:39.804750 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.804777 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 21:16:39.804904 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40319: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40319: connect: Connection refused (error 111)
I20260927 21:16:39.820641 22658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:39.822686 22658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:39.822803 22658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:39.826740 22658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40319
I20260927 21:16:39.826828 22725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40319 every 8 connection(s)
I20260927 21:16:39.827131 22658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:16:39.828109 22726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:39.829761 22726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:39.835270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 22658
I20260927 21:16:39.835386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-1/wal/instance
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:39.835573 392 external_mini_cluster.cc:1428] Setting key 89042a269737c6820ad1b715c42479bb
I20260927 21:16:39.839488 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.830494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.0.98.62:41405)
0927 21:16:39.830553 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:39.830555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:39.830770 (+ 215us) server_negotiation.cc:407] Connection header received
0927 21:16:39.831279 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:39.831280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:39.831296 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:39.831349 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:39.831984 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.832457 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.833445 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.833640 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.833704 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.836164 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:39.836165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:39.836166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:39.836173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:39.837765 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.838294 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.838295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.838296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:39.838367 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.838709 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.838710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.838711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:39.838859 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:39.838931 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:39.839287 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 21:16:39.839384 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:16:39.839633 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.830451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41405 (local address 127.0.0.1:33328)
0927 21:16:39.830639 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.830677 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:39.831161 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:39.831426 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:39.831429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:39.831684 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:39.831887 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.831892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.832541 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.832542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.833358 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.833361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.833737 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.833738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.833848 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.834342 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:39.834351 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:39.836047 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:39.837975 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.837982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.837984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.838228 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:39.838454 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.838455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.838455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.838613 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:39.838946 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:39.838953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:39.839143 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 21:16:39.839245 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:16:39.839386 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260927 21:16:39.840247 22726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:16:39.849797 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.840543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40319 (local address 127.0.0.1:49362)
0927 21:16:39.840639 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.840651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:39.840696 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:39.841041 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:39.841042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:39.841199 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:39.841298 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.841300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.842166 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.842166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.842974 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.842977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.843485 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.843486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.843543 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.843893 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:39.843897 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:39.846574 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:39.848390 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.848392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.848392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.848593 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:39.848895 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.848896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.848897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.848956 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:39.849373 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:39.849374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:39.849429 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 21:16:39.849504 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:16:39.849622 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:16:39.849807 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.840692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.0.98.61:40319)
0927 21:16:39.840893 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:39.840896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:39.840916 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:16:39.840934 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:39.840936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:39.840947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:39.841017 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:39.841366 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.842086 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.843058 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.843413 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.843468 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.846673 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:39.846686 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:39.846686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:39.846696 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:39.848277 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.848697 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.848700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.848700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:39.848763 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.849058 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.849059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.849059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:39.849302 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:39.849385 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:39.849547 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:16:39.849689 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":171,"thread_start_us":104,"threads_started":1}
I20260927 21:16:39.850656 22726 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:39.851013 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.850874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58594)
0927 21:16:39.850925 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.850984 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:16:39.851161 22676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
I20260927 21:16:39.851574 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.851474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58598)
0927 21:16:39.851519 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.851551 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 21:16:39.851747 22726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 21:16:39.860594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41835
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41405,127.0.98.61:40319,127.0.98.60:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:16:39.896095 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } attempt: 3
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:16:39.900954 22726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 1
I20260927 21:16:39.901425 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.901311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58610)
0927 21:16:39.901362 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.901391 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 21:16:39.901835 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.901728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58612)
0927 21:16:39.901773 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.901806 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:16:39.901908 22726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
I20260927 21:16:39.902940 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.896662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.0.98.61:40319)
0927 21:16:39.896723 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:39.896725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:39.896733 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:39.896760 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:39.896761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:39.896770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:39.896815 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:39.897142 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.897671 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.898336 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.898541 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.898598 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.900456 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:39.900458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:39.900458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:39.900466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:39.901927 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.902241 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.902242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.902242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:39.902292 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.902455 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.902455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.902456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:39.902580 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:39.902637 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:39.902774 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 21:16:39.902851 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:16:39.902944 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.896578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40319 (local address 127.0.0.1:49364)
0927 21:16:39.896641 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.896651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:39.896687 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:39.896831 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:39.896832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:39.896997 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:39.897062 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.897064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.897756 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.897757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.898260 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.898262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.898606 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.898607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.898678 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.899087 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:39.899092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:39.900371 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:39.901994 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.901995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.901996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.902167 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:39.902343 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.902344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.902344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.902404 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:39.902634 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:39.902635 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:39.902684 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 21:16:39.902744 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:16:39.902851 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:16:39.903420 22651 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:39.903755 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.903661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58628)
0927 21:16:39.903703 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.903728 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 21:16:39.904043 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.903982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58630)
0927 21:16:39.904011 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.904031 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 21:16:39.904124 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
I20260927 21:16:39.935280 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 1
I20260927 21:16:39.935824 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.935668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58638)
0927 21:16:39.935746 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.935795 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:16:39.936203 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.936100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58654)
0927 21:16:39.936157 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.936183 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 21:16:39.936298 22651 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
W20260927 21:16:39.944064 22733 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:16:39.944298 22733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:39.944314 22733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:39.944334 22733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:39.945534 22733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:39.945580 22733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:39.945592 22733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:39.945611 22733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:39.945621 22733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:39.947106 22733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41405,127.0.98.61:40319,127.0.98.60:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:39.947316 22733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:39.947494 22733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:39.952472 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:39.952553 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:39.952553 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:39.952739 22733 server_base.cc:1034] running on GCE node
I20260927 21:16:39.952973 22733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:39.953299 22733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:39.954486 22733 hybrid_clock.cc:648] HybridClock initialized: now 1790543799954470 us; error 31 us; skew 500 ppm
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:39.956935 22733 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 21:16:39.957443 22733 webserver.cc:460] Webserver started at http://127.0.98.60:42675/ using document root <none> and password file <none>
I20260927 21:16:39.957630 22733 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:39.957671 22733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:39.957757 22733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:39.958775 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/data/instance:
uuid: "443f884ed901411cbc91ed022a6c7d50"
format_stamp: "Formatted at 2026-09-27 21:16:39 on dist-test-slave-r9x1"
server_key: "f5688331837f5fd3a77e4b5d82c115de"
server_key_iv: "b7d8dcc58da09ec9f3ca0f93f729ea2b"
server_key_version: "encryptionkey@0"
I20260927 21:16:39.959080 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/wal/instance:
uuid: "443f884ed901411cbc91ed022a6c7d50"
format_stamp: "Formatted at 2026-09-27 21:16:39 on dist-test-slave-r9x1"
server_key: "f5688331837f5fd3a77e4b5d82c115de"
server_key_iv: "b7d8dcc58da09ec9f3ca0f93f729ea2b"
server_key_version: "encryptionkey@0"
I20260927 21:16:39.960428 22733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:16:39.961172 22748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:39.961395 22733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:16:39.961484 22733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/wal
uuid: "443f884ed901411cbc91ed022a6c7d50"
format_stamp: "Formatted at 2026-09-27 21:16:39 on dist-test-slave-r9x1"
server_key: "f5688331837f5fd3a77e4b5d82c115de"
server_key_iv: "b7d8dcc58da09ec9f3ca0f93f729ea2b"
server_key_version: "encryptionkey@0"
I20260927 21:16:39.961553 22733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:39.970052 22726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 2
I20260927 21:16:39.970530 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.970408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58666)
0927 21:16:39.970464 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.970497 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 21:16:39.970875 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.970801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58674)
0927 21:16:39.970835 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.970858 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 21:16:39.970952 22726 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
I20260927 21:16:39.979542 22733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:39.982041 22733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:39.982266 22733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:39.987639 22733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41835
I20260927 21:16:39.987753 22800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41835 every 8 connection(s)
I20260927 21:16:39.988071 22733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:16:39.989243 22801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:39.990968 22801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:39.995682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 22733
I20260927 21:16:39.995796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543529727471-392-0/minicluster-data/master-2/wal/instance
I20260927 21:16:39.995971 392 external_mini_cluster.cc:1428] Setting key df42a91ba95575f98d546177a8eb3ff4
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:40.000728 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.991866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.0.98.62:41405)
0927 21:16:39.991922 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:39.991924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:39.992192 (+ 268us) server_negotiation.cc:407] Connection header received
0927 21:16:39.992710 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:39.992711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:39.992722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:39.992777 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:39.993431 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.993894 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.994956 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.995130 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.995176 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.997658 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:39.997660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:39.997661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:39.997669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:39.999292 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.999678 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.999679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.999680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:39.999741 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:39.999979 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:39.999980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:39.999981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.000110 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.000162 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.000588 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.000659 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:16:40.000941 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.991886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41405 (local address 127.0.0.1:33344)
0927 21:16:39.992047 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:39.992084 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:39.992584 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:39.992833 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:39.992837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:39.993105 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:39.993359 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.993364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.993993 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.993994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.994885 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:39.994888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.995251 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.995252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:39.995362 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:39.995958 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:39.995975 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:39.997572 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:39.999364 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.999372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.999374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.999605 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:39.999795 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:39.999795 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:39.999796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:39.999910 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:40.000177 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:40.000185 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:40.000360 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 21:16:40.000500 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 21:16:40.000689 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260927 21:16:40.001514 22801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 21:16:39 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:16:40.005327 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:39.996352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.0.98.60:41835)
0927 21:16:39.996660 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:39.996663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:39.996678 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:16:39.996696 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:39.996698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:39.996722 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:39.996781 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:39.997430 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.998135 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.999233 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:39.999569 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:39.999643 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.002288 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.002294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.002295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.002301 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.004044 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.004428 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.004431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.004431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.004491 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.004680 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.004681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.004681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.004818 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.004972 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.005052 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.005131 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":89,"threads_started":1}
I20260927 21:16:40.008728 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.001873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.0.98.61:40319)
0927 21:16:40.001921 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.001923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.001966 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:16:40.002039 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.002039 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.002048 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.002115 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.002366 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.002980 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.003627 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.003844 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.003898 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.005628 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.005630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.005630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.005639 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.007710 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.008092 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.008093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.008093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.008145 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.008316 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.008316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.008317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.008413 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.008456 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.008585 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.008640 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:16:40.008759 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.001804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40319 (local address 127.0.0.1:49366)
0927 21:16:40.001921 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:40.001932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:40.001980 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:40.002100 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:40.002101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:40.002244 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:40.002306 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:40.002308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.003053 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.003054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:40.003545 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:40.003546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.003901 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.003902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:40.003973 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.004320 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:40.004325 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:40.005553 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:40.007807 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:40.007808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:40.007809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:40.008024 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:40.008199 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:40.008199 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:40.008200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:40.008263 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:40.008458 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:40.008458 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:40.008501 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:16:40.008567 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:16:40.008662 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 21:16:40.009130 22801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:16:40 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:40.011468 22651 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 2
Sep 27 21:16:40 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:16:40.015618 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.006255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.0.98.62:41405)
0927 21:16:40.006310 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.006312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.006343 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:16:40.006394 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.006395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.006406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.006455 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.007040 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.007787 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.008736 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.008998 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.009064 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.011531 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.011533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.011534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.011543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.013792 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.014410 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.014411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.014411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.014476 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.014913 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.014914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.014914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.015038 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.015218 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.015434 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.015519 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 21:16:40 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:16:40.017006 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.009499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.0.98.60:41835)
0927 21:16:40.009540 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.009541 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.009549 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:40.009571 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.009571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.009582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.009670 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.009970 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.010644 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.011453 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.011781 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.011847 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.013663 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.013664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.013665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.013671 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.015645 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.015965 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.015967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.015967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.016027 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.016279 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.016280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.016280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.016410 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.016467 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.016610 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.016907 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:16:40.017501 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.009438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58696)
0927 21:16:40.009483 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:40.009493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:40.009528 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:40.009725 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:40.009726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:40.009868 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:40.009901 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:40.009903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.010737 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.010738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:40.011377 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:40.011379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.011865 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.011866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:40.011929 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.012351 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:40.012356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:40.013588 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:40.015717 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:40.015719 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:40.015719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:40.015908 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:40.016156 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:40.016157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:40.016157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:40.016211 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:40.016469 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:40.016470 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:40.016509 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 21:16:40.017081 (+ 572us) client_negotiation.cc:240] Negotiation successful
0927 21:16:40.017411 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 21:16:40.019594 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.011727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58704)
0927 21:16:40.011788 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:40.011800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:40.011837 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:40.012194 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:40.012195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:40.012339 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:40.012381 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:40.012382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.013399 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.013400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:40.014100 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:40.014102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.014430 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.014431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:40.014486 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.014814 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:40.014818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:40.016158 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:40.018509 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:40.018511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:40.018512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:40.018729 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:40.018938 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:40.018938 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:40.018939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:40.018994 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:40.019297 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:40.019298 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:40.019346 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:16:40.019390 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 21:16:40.019490 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:16:40.019913 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50: Bootstrap starting.
I20260927 21:16:40.020155 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.011914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.0.98.60:41835)
0927 21:16:40.012060 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.012061 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.012068 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:16:40.012087 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.012088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.012098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.012162 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.012468 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.013170 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.014171 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.014363 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.014615 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.016242 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.016243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.016243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.016249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.018411 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.018815 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.018816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.018816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.018873 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.019050 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.019051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.019051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.019221 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.019869 (+ 648us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.019912 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.020022 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260927 21:16:40.020695 22801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:40.020963 22801 log.cc:824] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:40.021680 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048: Bootstrap starting.
Sep 27 21:16:40 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:16:40.022233 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50: No bootstrap required, opened a new log
I20260927 21:16:40.022260 22651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:40.022563 22651 log.cc:824] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:40.023609 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048: No bootstrap required, opened a new log
I20260927 21:16:40.023957 22801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 21:16:40.024145 22801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443f884ed901411cbc91ed022a6c7d50, State: Initialized, Role: FOLLOWER
I20260927 21:16:40.024242 22801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 21:16:40.024513 22809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.024569 22801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:40.024611 22809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:40.025713 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.016314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.0.98.61:40319)
0927 21:16:40.016353 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.016355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.016364 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:40.016640 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.016641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.016653 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.016683 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.017317 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.018401 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.019334 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.019639 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.019684 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.022292 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.022294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.022294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.022302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.023928 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.024342 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.024344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.024344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.024411 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.024799 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.024800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.024801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.024941 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.025096 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.025538 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.025630 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:16:40.025708 22651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 21:16:40.025846 22651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973669ddb82b4b16837d283f79f3b048, State: Initialized, Role: FOLLOWER
I20260927 21:16:40.025915 22651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 21:16:40.026453 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.026599 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:16:40.026309 22820 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 21:16:40.026897 22651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:40.027320 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:40.028280 22625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "973669ddb82b4b16837d283f79f3b048"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33348
I20260927 21:16:40.028576 22625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:16:40.028626 22625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:16:40.028416 22820 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:40.026832 22733 master_runner.cc:186] Error getting master registration for 127.0.98.62:41405: OK, instance_id { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" instance_seqno: 1790543799667547 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:16:40.029564 22733 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:16:40.029625 22625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 21:16:40.029913 22625 leader_election.cc:288] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [CANDIDATE]: Term 1 election: Requested vote from peers e1fa9783664d44be99a935283f5b75c4 (127.0.98.61:40319), 443f884ed901411cbc91ed022a6c7d50 (127.0.98.60:41835)
W20260927 21:16:40.030479 22833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:16:40.030555 22833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:40.031173 22773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "973669ddb82b4b16837d283f79f3b048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "443f884ed901411cbc91ed022a6c7d50"
I20260927 21:16:40.031391 22773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:40.032392 22773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 973669ddb82b4b16837d283f79f3b048 in term 1.
I20260927 21:16:40.032666 22601 leader_election.cc:302] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443f884ed901411cbc91ed022a6c7d50, 973669ddb82b4b16837d283f79f3b048; no voters:
I20260927 21:16:40.032783 22821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:40.032917 22821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [term 1 LEADER]: Becoming Leader. State: Replica: 973669ddb82b4b16837d283f79f3b048, State: Running, Role: LEADER
I20260927 21:16:40.033015 22821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 21:16:40.033479 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973669ddb82b4b16837d283f79f3b048. Latest consensus state: current_term: 1 leader_uuid: "973669ddb82b4b16837d283f79f3b048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.033566 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:40.032608 22583 master_runner.cc:186] Error getting master registration for 127.0.98.61:40319: OK, instance_id { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" instance_seqno: 1790543799825598 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:16:40.035000 22583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:16:40.039718 22842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:40.039947 22842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:40.046543 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.031194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.0.98.61:40319)
0927 21:16:40.031318 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.031320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.033434 (+ 2114us) server_negotiation.cc:407] Connection header received
0927 21:16:40.033560 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.033561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.033575 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.033608 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.034461 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.035160 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.038358 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.038699 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.038772 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.042769 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.042771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.042771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.042782 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.044730 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.045144 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.045145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.045145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.045208 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.045662 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.045663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.045663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.045816 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.046138 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.046354 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.046440 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 21:16:40.048458 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.038598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.0.98.62:41405)
0927 21:16:40.038645 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.038647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.039089 (+ 442us) server_negotiation.cc:407] Connection header received
0927 21:16:40.039224 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.039238 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.039251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.039284 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.040550 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.041329 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.042778 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.043050 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.043104 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.045171 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.045173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.045173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.045180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.046696 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.047333 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.047335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.047335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.047409 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.047711 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.047712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.047712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.047861 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.047959 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.048278 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.048363 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:16:40.050446 22773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [term 1 FOLLOWER]: Refusing update from remote peer 973669ddb82b4b16837d283f79f3b048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:40.051152 22840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:40.053016 22809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973669ddb82b4b16837d283f79f3b048. Latest consensus state: current_term: 1 leader_uuid: "973669ddb82b4b16837d283f79f3b048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.053133 22809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:40.054239 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.034649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.0.98.60:41835)
0927 21:16:40.035080 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.035082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.039587 (+ 4505us) server_negotiation.cc:407] Connection header received
0927 21:16:40.039619 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.039620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.039631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.039686 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.041064 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.041781 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.046992 (+ 5211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.047347 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.047405 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.049536 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.049538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.049538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.049563 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.052339 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.053056 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.053057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.053058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.053118 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.053382 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.053383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.053383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.053635 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.053777 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.053940 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.054106 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405}
I20260927 21:16:40.062106 22726 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 3
I20260927 21:16:40.064565 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "973669ddb82b4b16837d283f79f3b048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.064683 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:40.065444 22809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973669ddb82b4b16837d283f79f3b048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.065551 22809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 21:16:40 dist-test-slave-r9x1 krb5kdc[22570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:16:40.070086 22842 catalog_manager.cc:1347] Generated new cluster ID: a0ea7f61b00244c0a72d6a09c65022f8
I20260927 21:16:40.070154 22842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:40.070767 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.062749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.0.98.60:41835)
0927 21:16:40.063151 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.063153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.063161 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:40.063180 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.063181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.063190 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.063626 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.063663 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.064337 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.065323 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.065604 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.065929 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.067824 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:40.067826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:40.067826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:40.067833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:40.069477 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.069940 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.069941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.069941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.069999 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:40.070220 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:40.070221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:40.070222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:40.070378 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:40.070437 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:40.070605 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 21:16:40.070674 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260927 21:16:40.070768 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:40.062455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:58720)
0927 21:16:40.062587 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:40.062599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:40.062637 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:40.063276 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:40.063277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:40.063471 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:40.063540 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:40.063542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.064441 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.064443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:40.065190 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:40.065194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:40.065687 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:40.065688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:40.065749 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:40.066224 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:40.066229 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:40.067720 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:40.069565 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:40.069566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:40.069566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:40.069800 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:40.070059 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:40.070078 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:40.070078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:40.070142 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:40.070436 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:40.070437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:40.070487 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:16:40.070553 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:16:40.070667 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 21:16:40.072464 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4: Bootstrap starting.
I20260927 21:16:40.073235 22726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:40.073745 22726 log.cc:824] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:40.089116 22726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4: No bootstrap required, opened a new log
I20260927 21:16:40.091792 22726 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 21:16:40.092031 22726 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fa9783664d44be99a935283f5b75c4, State: Initialized, Role: FOLLOWER
I20260927 21:16:40.092142 22726 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 21:16:40.092507 22852 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.092623 22852 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:40.093430 22699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:40.094475 22699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [term 1 FOLLOWER]: Refusing update from remote peer 973669ddb82b4b16837d283f79f3b048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:40.094907 22840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:40.095348 22700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "973669ddb82b4b16837d283f79f3b048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e1fa9783664d44be99a935283f5b75c4"
I20260927 21:16:40.092569 22726 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:40.097481 22842 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:40.098279 22842 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:16:40.100008 22852 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 973669ddb82b4b16837d283f79f3b048. Latest consensus state: current_term: 1 leader_uuid: "973669ddb82b4b16837d283f79f3b048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.100220 22852 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:40.100298 22852 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "973669ddb82b4b16837d283f79f3b048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.100391 22852 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:40.101866 22853 mvcc.cc:205] Tried to move back new op lower bound from 7334067405001187328 to 7334067404935987200. Current Snapshot: MvccSnapshot[applied={T|T < 7334067405001187328}]
I20260927 21:16:40.102710 22700 raft_consensus.cc:2396] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 973669ddb82b4b16837d283f79f3b048 for term 1 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 21:16:40.103484 22840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "973669ddb82b4b16837d283f79f3b048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "973669ddb82b4b16837d283f79f3b048" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41405 } } peers { permanent_uuid: "e1fa9783664d44be99a935283f5b75c4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40319 } } peers { permanent_uuid: "443f884ed901411cbc91ed022a6c7d50" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 21:16:40.103581 22840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048 [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:40.106140 22866 catalog_manager.cc:1259] Loaded cluster ID: a0ea7f61b00244c0a72d6a09c65022f8
I20260927 21:16:40.106209 22866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4: loading cluster ID for follower catalog manager: success
I20260927 21:16:40.107364 22866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4: acquiring CA information for follower catalog manager: success
W20260927 21:16:40.107483 22866 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260927 21:16:40.115340 22842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 973669ddb82b4b16837d283f79f3b048: Generated new TSK 0
I20260927 21:16:40.115576 22842 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:16:40.148260 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.146755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.0.98.61:40319)
0927 21:16:40.146801 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.146803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.147487 (+ 684us) server_negotiation.cc:407] Connection header received
0927 21:16:40.147611 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.147613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.147625 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.147656 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.148163 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49400: BlockingRecv error: recv got EOF from 127.0.0.1:49400 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:16:40.149315 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.147172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.0.98.60:41835)
0927 21:16:40.147214 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.147215 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.148727 (+ 1512us) server_negotiation.cc:407] Connection header received
0927 21:16:40.148825 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.148826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.148838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.148867 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.149224 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58726: BlockingRecv error: recv got EOF from 127.0.0.1:58726 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:16:40.150568 22653 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.149888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.0.98.62:41405)
0927 21:16:40.149932 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.149935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.150029 (+ 94us) server_negotiation.cc:407] Connection header received
0927 21:16:40.150121 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.150122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.150132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.150161 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.150490 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33366: BlockingRecv error: recv got EOF from 127.0.0.1:33366 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 21:16:40.153156 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.152328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.0.98.61:40319)
0927 21:16:40.152367 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.152369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.152626 (+ 257us) server_negotiation.cc:407] Connection header received
0927 21:16:40.152733 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.152734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.152754 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.152781 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.153104 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49404: BlockingRecv error: recv got EOF from 127.0.0.1:49404 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 21:16:40.156481 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.155643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.0.98.61:40319)
0927 21:16:40.155682 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.155684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.155944 (+ 260us) server_negotiation.cc:407] Connection header received
0927 21:16:40.156038 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.156039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.156050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.156077 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.156422 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49418: BlockingRecv error: recv got EOF from 127.0.0.1:49418 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:16:40.160866 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.159902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.0.98.61:40319)
0927 21:16:40.159955 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.159957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.160246 (+ 289us) server_negotiation.cc:407] Connection header received
0927 21:16:40.160352 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.160354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.160366 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.160396 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.160806 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49424: BlockingRecv error: recv got EOF from 127.0.0.1:49424 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:16:40.167559 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.166563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.0.98.61:40319)
0927 21:16:40.166611 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.166613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.166928 (+ 315us) server_negotiation.cc:407] Connection header received
0927 21:16:40.167036 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.167037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.167051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.167082 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.167483 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49426: BlockingRecv error: recv got EOF from 127.0.0.1:49426 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 21:16:40.178189 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.177108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.0.98.61:40319)
0927 21:16:40.177171 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.177173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.177537 (+ 364us) server_negotiation.cc:407] Connection header received
0927 21:16:40.177648 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.177650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.177662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.177691 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.178122 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49434: BlockingRecv error: recv got EOF from 127.0.0.1:49434 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 21:16:40.196606 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.195627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.0.98.61:40319)
0927 21:16:40.195675 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.195677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.195978 (+ 301us) server_negotiation.cc:407] Connection header received
0927 21:16:40.196103 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.196104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.196120 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.196150 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.196544 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49440: BlockingRecv error: recv got EOF from 127.0.0.1:49440 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:16:40.230751 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.230043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.0.98.61:40319)
0927 21:16:40.230107 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.230109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.230233 (+ 124us) server_negotiation.cc:407] Connection header received
0927 21:16:40.230295 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.230296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.230308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.230358 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.230689 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49442: BlockingRecv error: recv got EOF from 127.0.0.1:49442 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 21:16:40.296883 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.296055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.0.98.61:40319)
0927 21:16:40.296131 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.296132 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.296192 (+ 60us) server_negotiation.cc:407] Connection header received
0927 21:16:40.296337 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.296339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.296351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.296409 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.296818 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49452: BlockingRecv error: recv got EOF from 127.0.0.1:49452 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 21:16:40.426751 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.426041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.0.98.61:40319)
0927 21:16:40.426102 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.426104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.426207 (+ 103us) server_negotiation.cc:407] Connection header received
0927 21:16:40.426276 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.426276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.426288 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.426338 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.426696 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49462: BlockingRecv error: recv got EOF from 127.0.0.1:49462 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 21:16:40.684521 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:40.683911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.0.98.61:40319)
0927 21:16:40.683955 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:40.683958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:40.683996 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:16:40.684068 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:40.684069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:40.684094 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:40.684141 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:40.684471 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59908: BlockingRecv error: recv got EOF from 127.0.0.1:59908 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:16:41.030155 22820 catalog_manager.cc:1259] Loaded cluster ID: a0ea7f61b00244c0a72d6a09c65022f8
I20260927 21:16:41.030237 22820 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50: loading cluster ID for follower catalog manager: success
I20260927 21:16:41.031121 22820 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50: acquiring CA information for follower catalog manager: success
I20260927 21:16:41.031417 22820 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 443f884ed901411cbc91ed022a6c7d50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:16:41.108106 22866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1fa9783664d44be99a935283f5b75c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:16:41.200389 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:41.197791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.0.98.61:40319)
0927 21:16:41.197904 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:41.197906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:41.197946 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:16:41.197970 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:41.197971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:41.198020 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:41.198087 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:41.198272 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.198781 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.199595 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.199823 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.199880 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:41.200064 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:41.200181 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:41.200238 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:41.200268 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 21:16:41.200291 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260927 21:16:41.204841 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:41.202316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.0.98.60:41835)
0927 21:16:41.202427 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:41.202429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:41.202438 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:41.202485 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:41.202485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:41.202520 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:41.202589 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:41.202703 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.203198 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.203934 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.204152 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.204215 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:41.204424 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:41.204541 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:41.204653 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:41.204722 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:16:41.204743 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260927 21:16:41.209497 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:41.206608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.0.98.62:41405)
0927 21:16:41.206734 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:41.206736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:41.206744 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:41.206814 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:41.206815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:41.206868 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:41.206952 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:41.207073 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.207821 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.208604 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.208887 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.208952 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:41.209126 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:41.209250 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:41.209329 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:41.209376 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 21:16:41.209401 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260927 21:16:41.209496 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:41.206650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.0.98.60:41835)
0927 21:16:41.206694 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:41.206695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:41.206747 (+ 52us) server_negotiation.cc:407] Connection header received
0927 21:16:41.206811 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:41.206811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:41.206823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:41.206866 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:41.207044 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.207797 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.208744 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.208964 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.209023 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:41.209155 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:41.209185 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:41.209233 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:41.209391 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 21:16:41.209421 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:16:41.209528 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:41.206479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.0.98.61:40319)
0927 21:16:41.206611 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:41.206612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:41.206856 (+ 244us) server_negotiation.cc:407] Connection header received
0927 21:16:41.206909 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:41.206910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:41.206921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:41.206966 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:41.207103 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.207582 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.208685 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:41.208965 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:41.209029 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:41.209254 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:41.209314 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:41.209353 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:41.209436 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 21:16:41.209458 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 21:16:41.211300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 22583
I20260927 21:16:41.216231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 22658
I20260927 21:16:41.220927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 22733
2026-09-27T21:16:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2750 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:16:41 dist-test-slave-r9x1 krb5kdc[22951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:16:41 dist-test-slave-r9x1 krb5kdc[22951](info): set up 2 sockets
Sep 27 21:16:41 dist-test-slave-r9x1 krb5kdc[22951](info): commencing operation
krb5kdc: starting...
W20260927 21:16:42.189694 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/test-user.keytab.
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:16:42.316222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34847,127.0.98.61:34049,127.0.98.60:35443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:16:42.397511 22964 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:16:42.397755 22964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:42.397773 22964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:42.397794 22964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:42.399300 22964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:42.399379 22964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:42.399394 22964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:42.399405 22964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:42.399416 22964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:42.401203 22964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34847,127.0.98.61:34049,127.0.98.60:35443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:42.401527 22964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:42.401769 22964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:42.406852 22969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:42.406821 22970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:42.406877 22964 server_base.cc:1034] running on GCE node
W20260927 21:16:42.406843 22972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:42.407411 22964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:42.407747 22964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:42.408931 22964 hybrid_clock.cc:648] HybridClock initialized: now 1790543802408911 us; error 36 us; skew 500 ppm
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:42.411775 22964 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:16:42.412293 22964 webserver.cc:460] Webserver started at http://127.0.98.62:33973/ using document root <none> and password file <none>
I20260927 21:16:42.412497 22964 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:42.412547 22964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:42.412678 22964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:42.413832 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/data/instance:
uuid: "a3895d31aec94c22af5f1faa44bd4a2e"
format_stamp: "Formatted at 2026-09-27 21:16:42 on dist-test-slave-r9x1"
server_key: "4dd1c8bd1e947dec16682731ed43263b"
server_key_iv: "9e640797ea185dc4f5cf9843c394deca"
server_key_version: "encryptionkey@0"
I20260927 21:16:42.414177 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3895d31aec94c22af5f1faa44bd4a2e"
format_stamp: "Formatted at 2026-09-27 21:16:42 on dist-test-slave-r9x1"
server_key: "4dd1c8bd1e947dec16682731ed43263b"
server_key_iv: "9e640797ea185dc4f5cf9843c394deca"
server_key_version: "encryptionkey@0"
I20260927 21:16:42.415467 22964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:16:42.416237 22979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:42.416392 22964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:16:42.416476 22964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/wal
uuid: "a3895d31aec94c22af5f1faa44bd4a2e"
format_stamp: "Formatted at 2026-09-27 21:16:42 on dist-test-slave-r9x1"
server_key: "4dd1c8bd1e947dec16682731ed43263b"
server_key_iv: "9e640797ea185dc4f5cf9843c394deca"
server_key_version: "encryptionkey@0"
I20260927 21:16:42.416544 22964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:42.439474 22964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:42.441490 22964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:42.441632 22964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:42.445361 22964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34847
I20260927 21:16:42.445405 23031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34847 every 8 connection(s)
I20260927 21:16:42.445737 22964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:16:42.446700 23032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:42.448256 23032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:42.451249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 22964
I20260927 21:16:42.451325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-0/wal/instance
I20260927 21:16:42.451496 392 external_mini_cluster.cc:1428] Setting key 67fbe29734be57c63c420d1bc7690c11
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:42.458640 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.448956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.0.1:56682)
0927 21:16:42.449138 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.449176 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:42.449722 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:42.450013 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:42.450016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:42.450281 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:42.450442 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.450446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.451530 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.451530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.452256 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.452259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.452756 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.452757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.452819 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.453343 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:42.453352 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:42.454833 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:42.457021 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.457029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.457031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.457246 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.457551 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.457552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.457552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.457651 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.458122 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:42.458128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:42.458194 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 21:16:42.458269 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:16:42.458427 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260927 21:16:42.458647 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.449135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.0.98.62:34847)
0927 21:16:42.449324 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.449329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.449361 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:16:42.449894 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.449896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.449914 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.450018 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.450560 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.451434 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.452340 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.452657 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.452755 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.454934 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.454944 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.454945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.454953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.456910 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.457356 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.457359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.457360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.457476 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.457722 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.457723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.457724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.458037 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.458136 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.458332 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.458506 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
I20260927 21:16:42.459456 23032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 21:16:42.459918 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.459738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34049 (local address 127.0.0.1:43410)
0927 21:16:42.459784 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.459889 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 21:16:42.460074 22981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
I20260927 21:16:42.460506 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.460404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34049 (local address 127.0.0.1:43424)
0927 21:16:42.460458 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.460486 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 21:16:42.460618 23032 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34049: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 21:16:42.476982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34049
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34847,127.0.98.61:34049,127.0.98.60:35443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:16:42.509881 23032 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } attempt: 1
I20260927 21:16:42.510457 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.510290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34049 (local address 127.0.0.1:43428)
0927 21:16:42.510378 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.510424 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:16:42.510784 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.510712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34049 (local address 127.0.0.1:43436)
0927 21:16:42.510746 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.510769 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 21:16:42.510870 23032 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34049: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
W20260927 21:16:42.558516 23039 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:16:42.558753 23039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:42.558769 23039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:42.558791 23039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:42.560024 23039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:42.560077 23039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:42.560098 23039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:42.560111 23039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:42.560122 23039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:42.561753 23039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34847,127.0.98.61:34049,127.0.98.60:35443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:42.561972 23039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:42.562165 23039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:42.567224 23044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:42.567207 23045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:42.567207 23047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:42.567507 23039 server_base.cc:1034] running on GCE node
I20260927 21:16:42.567777 23039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:42.568076 23039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:42.569226 23039 hybrid_clock.cc:648] HybridClock initialized: now 1790543802569202 us; error 37 us; skew 500 ppm
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:42.571885 23039 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 21:16:42.572396 23039 webserver.cc:460] Webserver started at http://127.0.98.61:39113/ using document root <none> and password file <none>
I20260927 21:16:42.572616 23039 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:42.572674 23039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:42.572801 23039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:42.573957 23039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/data/instance:
uuid: "93f2e85fa5264c6f9af1950c51238cb0"
format_stamp: "Formatted at 2026-09-27 21:16:42 on dist-test-slave-r9x1"
server_key: "28093d4808c3d80b9d0c95aff11c79ed"
server_key_iv: "10a41e8e2226d97fb1c8593165c729c8"
server_key_version: "encryptionkey@0"
I20260927 21:16:42.574304 23039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/wal/instance:
uuid: "93f2e85fa5264c6f9af1950c51238cb0"
format_stamp: "Formatted at 2026-09-27 21:16:42 on dist-test-slave-r9x1"
server_key: "28093d4808c3d80b9d0c95aff11c79ed"
server_key_iv: "10a41e8e2226d97fb1c8593165c729c8"
server_key_version: "encryptionkey@0"
I20260927 21:16:42.575618 23039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:16:42.576457 23054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:42.576717 23039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:16:42.576896 23039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/wal
uuid: "93f2e85fa5264c6f9af1950c51238cb0"
format_stamp: "Formatted at 2026-09-27 21:16:42 on dist-test-slave-r9x1"
server_key: "28093d4808c3d80b9d0c95aff11c79ed"
server_key_iv: "10a41e8e2226d97fb1c8593165c729c8"
server_key_version: "encryptionkey@0"
I20260927 21:16:42.576985 23039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:42.579058 23032 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } attempt: 2
I20260927 21:16:42.579574 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.579418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34049 (local address 127.0.0.1:43448)
0927 21:16:42.579488 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.579534 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 21:16:42.579973 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.579873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34049 (local address 127.0.0.1:43458)
0927 21:16:42.579921 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.579950 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:16:42.580070 23032 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34049: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34049: connect: Connection refused (error 111)
I20260927 21:16:42.605695 23039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:42.607599 23039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:42.607726 23039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:42.611642 23039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34049
I20260927 21:16:42.611687 23106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34049 every 8 connection(s)
I20260927 21:16:42.612020 23039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:16:42.612993 23107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:42.614593 23107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:42.622110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 23039
I20260927 21:16:42.622190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-1/wal/instance
I20260927 21:16:42.622375 392 external_mini_cluster.cc:1428] Setting key 0223176222e9f221b726bf85db3653c7
I20260927 21:16:42.624120 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.615334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.0.98.62:34847)
0927 21:16:42.615399 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.615401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.615580 (+ 179us) server_negotiation.cc:407] Connection header received
0927 21:16:42.616109 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.616109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.616123 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.616174 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.616859 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.617421 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.618475 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.618702 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.618763 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.621197 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.621199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.621200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.621207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.622606 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.623026 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.623027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.623028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.623087 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.623320 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.623321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.623321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.623468 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.623536 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.623922 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.624022 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:16:42.624315 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.615296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.0.1:56694)
0927 21:16:42.615463 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.615506 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:42.615986 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:42.616273 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:42.616277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:42.616541 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:42.616745 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.616749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.617512 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.617513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.618377 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.618380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.618781 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.618782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.618890 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.619475 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:42.619489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:42.621091 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:42.622697 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.622707 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.622710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.622954 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.623144 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.623145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.623145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.623263 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.623548 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:42.623555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:42.623703 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 21:16:42.623900 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 21:16:42.624073 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260927 21:16:42.624914 23107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 21:16:42.632148 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.625172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34049 (local address 127.0.0.1:43460)
0927 21:16:42.625217 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.625227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:42.625258 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:42.625534 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:42.625535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:42.625695 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:42.625758 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.625760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.626591 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.626592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.627092 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.627094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.627553 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.627554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.627625 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.628089 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:42.628094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:42.629445 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:42.631085 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.631086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.631087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.631251 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.631426 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.631426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.631427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.631477 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.631820 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:42.631821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:42.631861 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:16:42.631922 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:16:42.632037 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 21:16:42.632251 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.625251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.0.98.61:34049)
0927 21:16:42.625421 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.625425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.625440 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:16:42.625456 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.625457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.625468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.625523 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.625826 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.626517 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.627171 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.627485 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.627544 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.629522 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.629534 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.629534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.629541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.631001 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.631318 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.631322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.631322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.631370 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.631557 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.631558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.631558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.631758 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.631833 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.631958 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.632161 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":143,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 21:16:42.632803 23107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:42.633178 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.633039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59410)
0927 21:16:42.633082 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.633149 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 21:16:42.633385 23058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
I20260927 21:16:42.633811 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.633724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59420)
0927 21:16:42.633760 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.633788 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 21:16:42.633953 23107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35443: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 21:16:42.648898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
/tmp/dist-test-taskm7AuFZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35443
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34847,127.0.98.61:34049,127.0.98.60:35443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:16:42.671247 23032 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } attempt: 3
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:16:42.678593 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.671763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.0.98.61:34049)
0927 21:16:42.671825 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.671827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.671837 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:16:42.671883 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.671885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.671896 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.671938 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.672290 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.672795 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.673549 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.673751 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.673808 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.675602 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.675603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.675604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.675612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.677062 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.677581 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.677583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.677583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.677644 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.677818 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.677819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.677820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.678152 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.678245 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.678403 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.678488 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:16:42.678593 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.671709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34049 (local address 127.0.0.1:43466)
0927 21:16:42.671771 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.671782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:42.671829 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:42.671953 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:42.671954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:42.672124 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:42.672200 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.672202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.672881 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.672882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.673464 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.673466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.673814 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.673815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.673886 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.674312 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:42.674317 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:42.675515 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:42.677176 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.677178 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.677178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.677493 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.677703 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.677704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.677704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.677765 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.678247 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:42.678248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:42.678306 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 21:16:42.678369 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:16:42.678493 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 21:16:42.679127 23032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:42.679526 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.679419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59434)
0927 21:16:42.679466 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.679500 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 21:16:42.679852 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.679785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59444)
0927 21:16:42.679808 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.679832 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 21:16:42.679939 23032 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35443: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
I20260927 21:16:42.683189 23107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } attempt: 1
I20260927 21:16:42.683630 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.683511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59448)
0927 21:16:42.683563 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.683595 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 21:16:42.683982 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.683857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59450)
0927 21:16:42.683913 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.683954 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 21:16:42.684080 23107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35443: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
I20260927 21:16:42.711107 23032 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } attempt: 1
I20260927 21:16:42.711666 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.711485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59452)
0927 21:16:42.711588 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.711632 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:16:42.712046 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.711957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59466)
0927 21:16:42.711994 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.712020 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 21:16:42.712146 23032 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35443: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
W20260927 21:16:42.731225 23114 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:16:42.731449 23114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:16:42.731472 23114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:16:42.731498 23114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:16:42.732676 23114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:16:42.732717 23114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:16:42.732736 23114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:16:42.732752 23114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:16:42.732769 23114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:16:42.734324 23114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34847,127.0.98.61:34049,127.0.98.60:35443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:16:42.734571 23114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:16:42.734788 23114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:16:42.739593 23120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:16:42.739586 23122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:42.739732 23114 server_base.cc:1034] running on GCE node
W20260927 21:16:42.739586 23119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:16:42.740141 23114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:16:42.740407 23114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:16:42.741537 23114 hybrid_clock.cc:648] HybridClock initialized: now 1790543802741517 us; error 37 us; skew 500 ppm
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:16:42.744032 23114 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 21:16:42.744499 23114 webserver.cc:460] Webserver started at http://127.0.98.60:33547/ using document root <none> and password file <none>
I20260927 21:16:42.744714 23114 fs_manager.cc:359] Metadata directory not provided
I20260927 21:16:42.744760 23114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:16:42.744881 23114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:16:42.745970 23114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/data/instance:
uuid: "efdac131313e4599bd357031fa15363b"
format_stamp: "Formatted at 2026-09-27 21:16:42 on dist-test-slave-r9x1"
server_key: "9231d9f2e95f7511a725304d2e6dd4c7"
server_key_iv: "6e72bef0839fcafd42efa13e93f33f10"
server_key_version: "encryptionkey@0"
I20260927 21:16:42.746300 23114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/wal/instance:
uuid: "efdac131313e4599bd357031fa15363b"
format_stamp: "Formatted at 2026-09-27 21:16:42 on dist-test-slave-r9x1"
server_key: "9231d9f2e95f7511a725304d2e6dd4c7"
server_key_iv: "6e72bef0839fcafd42efa13e93f33f10"
server_key_version: "encryptionkey@0"
I20260927 21:16:42.747553 23114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:16:42.748219 23129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:16:42.748389 23114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:16:42.748476 23114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/wal
uuid: "efdac131313e4599bd357031fa15363b"
format_stamp: "Formatted at 2026-09-27 21:16:42 on dist-test-slave-r9x1"
server_key: "9231d9f2e95f7511a725304d2e6dd4c7"
server_key_iv: "6e72bef0839fcafd42efa13e93f33f10"
server_key_version: "encryptionkey@0"
I20260927 21:16:42.748538 23114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:16:42.752225 23107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } attempt: 2
I20260927 21:16:42.752661 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.752518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59468)
0927 21:16:42.752576 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.752622 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:16:42.753033 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.752957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59478)
0927 21:16:42.752995 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.753017 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 21:16:42.753181 23107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35443: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35443: connect: Connection refused (error 111)
I20260927 21:16:42.761703 23114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:16:42.763978 23114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:16:42.764124 23114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:16:42.768005 23114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35443
I20260927 21:16:42.768038 23181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35443 every 8 connection(s)
I20260927 21:16:42.768374 23114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:16:42.769487 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:16:42.770998 23182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:42.773728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu as pid 23114
I20260927 21:16:42.773821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm7AuFZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543529727471-392-0/minicluster-data/master-2/wal/instance
I20260927 21:16:42.773995 392 external_mini_cluster.cc:1428] Setting key b81bf3d8c3755f3b8d0f1a670447feed
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:16:42.780246 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.771734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.0.98.62:34847)
0927 21:16:42.771796 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.771798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.771986 (+ 188us) server_negotiation.cc:407] Connection header received
0927 21:16:42.772474 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.772475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.772488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.772540 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.773167 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.773681 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.774929 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.775129 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.775186 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.777456 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.777458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.777458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.777465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.778825 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.779221 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.779222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.779222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.779287 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.779556 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.779557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.779557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.779705 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.779762 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.780062 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.780150 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:16:42.780467 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.771712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.0.1:56696)
0927 21:16:42.771870 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.771906 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:42.772370 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:42.772623 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:42.772626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:42.772878 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:42.773078 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.773081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.773765 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.773765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.774858 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.774859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.775215 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.775216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.775357 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.775885 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:42.775897 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:42.777334 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:42.778912 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.778921 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.778923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.779152 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.779359 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.779360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.779360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.779491 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.779770 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:42.779778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:42.779917 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 21:16:42.780010 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:16:42.780192 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260927 21:16:42.780967 23182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:42.782562 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.774363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.0.98.60:35443)
0927 21:16:42.774527 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.774532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.774554 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:16:42.774598 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.774600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.774618 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.774692 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.775274 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.776168 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.776915 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.777303 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.777370 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.779652 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.779656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.779656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.779661 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.781336 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.781716 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.781719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.781720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.781783 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.781993 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.781993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.781994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.782160 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.782309 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.782354 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.782435 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:16:42.787287 23032 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } attempt: 2
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:16:42.787798 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.781342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.0.98.61:34049)
0927 21:16:42.781379 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.781381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.781388 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:16:42.781411 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.781412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.781421 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.781462 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.781710 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.782477 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.783288 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.783508 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.783567 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.785394 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.785396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.785396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.785404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.786658 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.786998 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.786999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.787000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.787054 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.787209 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.787210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.787210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.787346 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.787411 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.787626 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.787708 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:16:42.788035 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.781225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34049 (local address 127.0.0.1:43474)
0927 21:16:42.781294 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.781305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:42.781348 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:42.781468 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:42.781469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:42.781600 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:42.781643 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.781645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.782545 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.782547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.783221 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.783224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.783575 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.783576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.783639 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.784036 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:42.784041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:42.785313 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:42.786738 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.786739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.786740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.786929 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.787109 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.787110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.787110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.787160 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.787441 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:42.787442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:42.787489 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:16:42.787876 (+ 387us) client_negotiation.cc:240] Negotiation successful
0927 21:16:42.787963 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 21:16:42.788451 23182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:16:42.791025 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.783266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.0.98.62:34847)
0927 21:16:42.783312 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.783313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.783343 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:16:42.783409 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.783410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.783419 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.783469 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.783958 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.784632 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.785413 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.785689 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.785751 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.788064 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.788065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.788065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.788072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.789836 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.790227 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.790227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.790228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.790294 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.790472 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.790472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.790472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.790607 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.790789 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.790830 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.790931 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:16:42.795275 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.787529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59496)
0927 21:16:42.787619 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.787629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:42.787676 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:42.787825 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:42.787826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:42.787973 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:42.788014 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.788016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.789599 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.789600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.790100 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.790102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.790552 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.790553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.790611 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.791015 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:42.791019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:42.792344 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:42.793898 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.793899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.793900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.794108 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.794279 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.794279 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.794280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.794329 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.794575 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:42.794576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:42.794982 (+ 406us) client_negotiation.cc:769] Sending connection context
0927 21:16:42.795067 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:16:42.795166 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 21:16:42.795288 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.787575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.0.98.60:35443)
0927 21:16:42.787613 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.787615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.787670 (+ 55us) server_negotiation.cc:407] Connection header received
0927 21:16:42.787721 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.787722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.787735 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.787811 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.788654 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.789509 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.790180 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.790467 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.791051 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.792424 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.792425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.792425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.792431 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.793823 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.794173 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.794174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.794174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.794231 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.794382 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.794382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.794383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.794501 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.794559 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.795119 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.795193 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:16:42.796439 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.788682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59506)
0927 21:16:42.788811 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.788819 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:42.788846 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:42.789074 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:42.789075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:42.789389 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:42.789429 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.789431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.790292 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.790292 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.790922 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.790925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.791334 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.791335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.791382 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.791817 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:42.791822 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:42.793139 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:42.795546 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.795547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.795547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.795696 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.795861 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.795862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.795862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.795913 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.796149 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:42.796150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:42.796188 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:16:42.796247 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 21:16:42.796346 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:16:42.797092 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.788765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.0.98.60:35443)
0927 21:16:42.788942 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.788943 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.788949 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:16:42.788964 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.788965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.788973 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.789030 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.789516 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.790240 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.790997 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.791269 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.791329 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.793231 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.793232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.793233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.793238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.795476 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.795758 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.795758 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.795759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.795809 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.795961 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.795962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.795963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.796093 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.796148 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.796277 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.796961 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":599,"server-negotiator.queue_time_us":160,"thread_start_us":111,"threads_started":1}
I20260927 21:16:42.797437 23032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e: Bootstrap starting.
I20260927 21:16:42.798193 23032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:42.798496 23032 log.cc:824] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:42.798532 23182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b: Bootstrap starting.
I20260927 21:16:42.799108 23182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:42.799321 23182 log.cc:824] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:42.799719 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.791761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.0.98.61:34049)
0927 21:16:42.791805 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.791806 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.791813 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:16:42.791834 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.791835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.791844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.791890 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.792491 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.793196 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.794131 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.794405 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.794466 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.797393 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.797395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.797395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.797403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.798710 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.799022 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.799023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.799023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.799086 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.799252 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.799253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.799253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.799366 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.799518 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.799553 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.799631 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:16:42.800067 23182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b: No bootstrap required, opened a new log
I20260927 21:16:42.801606 23182 raft_consensus.cc:348] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } }
I20260927 21:16:42.801774 23182 raft_consensus.cc:729] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdac131313e4599bd357031fa15363b, State: Initialized, Role: FOLLOWER
I20260927 21:16:42.801858 23182 consensus_queue.cc:260] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } }
I20260927 21:16:42.802069 23032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e: No bootstrap required, opened a new log
I20260927 21:16:42.802111 23190 sys_catalog.cc:455] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:42.802184 23182 sys_catalog.cc:564] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:42.802205 23190 sys_catalog.cc:458] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:42.804109 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:16:42.804181 23032 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } }
I20260927 21:16:42.804323 23032 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3895d31aec94c22af5f1faa44bd4a2e, State: Initialized, Role: FOLLOWER
I20260927 21:16:42.804394 23032 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } }
I20260927 21:16:42.804662 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:42.804711 23032 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:42.804839 23006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a3895d31aec94c22af5f1faa44bd4a2e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56706
I20260927 21:16:42.804951 23006 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:16:42.804991 23006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:16:42.804039 23201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:16:42.805962 23201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:42.806090 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:42.806209 23006 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } }
I20260927 21:16:42.806739 23156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "efdac131313e4599bd357031fa15363b"
I20260927 21:16:42.806842 23156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:42.805395 23114 master_runner.cc:186] Error getting master registration for 127.0.98.62:34847: OK, instance_id { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" instance_seqno: 1790543802444284 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:16:42.807678 23114 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:16:42.807757 23156 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3895d31aec94c22af5f1faa44bd4a2e in term 1.
I20260927 21:16:42.806442 23006 leader_election.cc:288] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [CANDIDATE]: Term 1 election: Requested vote from peers 93f2e85fa5264c6f9af1950c51238cb0 (127.0.98.61:34049), efdac131313e4599bd357031fa15363b (127.0.98.60:35443)
I20260927 21:16:42.807976 22983 leader_election.cc:302] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3895d31aec94c22af5f1faa44bd4a2e, efdac131313e4599bd357031fa15363b; no voters:
I20260927 21:16:42.808103 23203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:16:42.808241 23203 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [term 1 LEADER]: Becoming Leader. State: Replica: a3895d31aec94c22af5f1faa44bd4a2e, State: Running, Role: LEADER
I20260927 21:16:42.808327 23203 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } }
I20260927 21:16:42.808785 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3895d31aec94c22af5f1faa44bd4a2e. Latest consensus state: current_term: 1 leader_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:42.808854 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:42.807991 22964 master_runner.cc:186] Error getting master registration for 127.0.98.61:34049: OK, instance_id { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" instance_seqno: 1790543802610483 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:16:42.810982 22964 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 21:16:42.815451 23214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:16:42.815516 23214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:42.817653 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.809011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.0.98.60:35443)
0927 21:16:42.809051 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.809054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.809203 (+ 149us) server_negotiation.cc:407] Connection header received
0927 21:16:42.809344 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.809345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.809357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.809387 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.810173 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.810930 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.811910 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.812230 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.812271 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.814315 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.814316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.814317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.814337 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.815899 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.816461 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.816463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.816463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.816536 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.816840 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.816841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.816841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.816988 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.817083 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.817453 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.817561 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:16:42.817442 23221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:16:42.819538 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.809339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.0.98.62:34847)
0927 21:16:42.809375 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.809377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.810101 (+ 724us) server_negotiation.cc:407] Connection header received
0927 21:16:42.810189 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.810190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.810202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.810228 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.811759 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.812444 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.813379 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.813685 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.813750 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.815971 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.815973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.815974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.815981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.818323 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.818717 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.818718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.818719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.818775 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.818953 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.818953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.818954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.819076 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.819158 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.819363 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.819444 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:16:42.820465 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.809606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.0.98.61:34049)
0927 21:16:42.809645 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.809646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.810931 (+ 1285us) server_negotiation.cc:407] Connection header received
0927 21:16:42.811022 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.811037 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.811048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.811077 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.811734 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.812446 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.814241 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.814525 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.814566 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.817049 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.817050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.817051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.817058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.818559 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.818924 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.818926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.818926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.818985 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.819219 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.819220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.819220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.819346 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.819885 (+ 539us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.820291 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.820374 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:16:42.820868 23221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:16:42.822690 23156 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [term 1 FOLLOWER]: Refusing update from remote peer a3895d31aec94c22af5f1faa44bd4a2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:42.823015 23215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:16:42.824394 23190 sys_catalog.cc:455] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3895d31aec94c22af5f1faa44bd4a2e. Latest consensus state: current_term: 1 leader_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:42.824496 23190 sys_catalog.cc:458] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:42.825636 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:42.825765 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [sys.catalog]: This master's current role is: LEADER
I20260927 21:16:42.825788 23190 sys_catalog.cc:455] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:42.825878 23190 sys_catalog.cc:458] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:42.826056 23226 mvcc.cc:205] Tried to move back new op lower bound from 7334067416358932480 to 7334067416303775744. Current Snapshot: MvccSnapshot[applied={T|T < 7334067416358932480}]
I20260927 21:16:42.826162 23227 mvcc.cc:205] Tried to move back new op lower bound from 7334067416358932480 to 7334067416303775744. Current Snapshot: MvccSnapshot[applied={T|T < 7334067416358932480}]
I20260927 21:16:42.826961 23221 catalog_manager.cc:1347] Generated new cluster ID: 55945055f53c49a6b35de404868541c3
I20260927 21:16:42.827018 23221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:16:42.844322 23107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } attempt: 3
I20260927 21:16:42.848438 23221 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:16:42.849213 23221 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 21:16:42 dist-test-slave-r9x1 krb5kdc[22951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:16:42.854341 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.844652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35443 (local address 127.0.0.1:59520)
0927 21:16:42.844744 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 21:16:42.844756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:16:42.844793 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:16:42.845039 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:16:42.845040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:16:42.845229 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:16:42.845860 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.845864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.847038 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.847039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.847864 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:16:42.847867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.848505 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.848507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:16:42.848642 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.849294 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:16:42.849300 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:16:42.850988 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:16:42.853077 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.853078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.853079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.853333 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.853597 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:16:42.853598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:16:42.853598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:16:42.853672 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:16:42.854003 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:16:42.854005 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:16:42.854069 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 21:16:42.854123 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:16:42.854234 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 21:16:42.854573 23221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e: Generated new TSK 0
I20260927 21:16:42.854827 23221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:16:42.855013 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:42.844866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59520 (local address 127.0.98.60:35443)
0927 21:16:42.844906 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.844908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.844915 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:16:42.844939 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.844940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.844950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.844993 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.845955 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.846704 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.848068 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:42.848379 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:42.848439 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:42.851131 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:16:42.851132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:16:42.851133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:16:42.851141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:16:42.852944 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.853429 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.853431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.853431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.853506 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:16:42.853749 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:16:42.853750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:16:42.853751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:16:42.853901 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:16:42.853958 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:42.854833 (+ 875us) server_negotiation.cc:299] Negotiation successful
0927 21:16:42.854920 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:16:42.857168 23107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0: Bootstrap starting.
I20260927 21:16:42.858096 23107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0: Neither blocks nor log segments found. Creating new log.
I20260927 21:16:42.858404 23107 log.cc:824] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:16:42.861563 23107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0: No bootstrap required, opened a new log
I20260927 21:16:42.863708 23107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } }
I20260927 21:16:42.863863 23107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f2e85fa5264c6f9af1950c51238cb0, State: Initialized, Role: FOLLOWER
I20260927 21:16:42.863941 23107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } }
I20260927 21:16:42.864276 23107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:16:42.864343 23080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:16:42.865394 23080 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [term 1 FOLLOWER]: Refusing update from remote peer a3895d31aec94c22af5f1faa44bd4a2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:16:42.865629 23081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "93f2e85fa5264c6f9af1950c51238cb0"
I20260927 21:16:42.866441 23081 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3895d31aec94c22af5f1faa44bd4a2e in term 1.
I20260927 21:16:42.866879 23215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:16:42.867429 23244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:16:42.867527 23244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:16:42.869558 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:42.869675 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:42.869720 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3895d31aec94c22af5f1faa44bd4a2e. Latest consensus state: current_term: 1 leader_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:42.869760 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:42.869782 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:42.869812 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:16:42.871790 23247 mvcc.cc:205] Tried to move back new op lower bound from 7334067416485920768 to 7334067416303775744. Current Snapshot: MvccSnapshot[applied={T|T < 7334067416358932480}]
W20260927 21:16:42.898329 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.893945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.0.98.60:35443)
0927 21:16:42.893984 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.893985 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.897578 (+ 3593us) server_negotiation.cc:407] Connection header received
0927 21:16:42.897703 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.897705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.897717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.897745 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.898253 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59522: BlockingRecv error: recv got EOF from 127.0.0.1:59522 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 21:16:42.899137 23035 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.893697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.0.98.62:34847)
0927 21:16:42.893748 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.893750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.898592 (+ 4842us) server_negotiation.cc:407] Connection header received
0927 21:16:42.898703 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.898705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.898717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.898744 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.899074 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56728: BlockingRecv error: recv got EOF from 127.0.0.1:56728 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 21:16:42.899873 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.894455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.0.98.61:34049)
0927 21:16:42.894495 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.894497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.899369 (+ 4872us) server_negotiation.cc:407] Connection header received
0927 21:16:42.899455 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.899456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.899467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.899496 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.899805 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43506: BlockingRecv error: recv got EOF from 127.0.0.1:43506 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:16:42.902583 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.901607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.0.98.61:34049)
0927 21:16:42.901647 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.901649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.901977 (+ 328us) server_negotiation.cc:407] Connection header received
0927 21:16:42.902093 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.902094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.902116 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.902145 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.902511 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43514: BlockingRecv error: recv got EOF from 127.0.0.1:43514 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:16:42.905879 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.904974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.0.98.61:34049)
0927 21:16:42.905012 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.905015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.905317 (+ 302us) server_negotiation.cc:407] Connection header received
0927 21:16:42.905413 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.905414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.905425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.905464 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.905821 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43524: BlockingRecv error: recv got EOF from 127.0.0.1:43524 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:16:42.910476 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.909513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.0.98.61:34049)
0927 21:16:42.909561 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.909563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.909861 (+ 298us) server_negotiation.cc:407] Connection header received
0927 21:16:42.909970 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.909971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.909985 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.910015 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.910401 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43536: BlockingRecv error: recv got EOF from 127.0.0.1:43536 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 21:16:42.916878 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.915871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.0.98.61:34049)
0927 21:16:42.915915 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.915917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.916240 (+ 323us) server_negotiation.cc:407] Connection header received
0927 21:16:42.916346 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.916348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.916361 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.916390 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.916814 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43540: BlockingRecv error: recv got EOF from 127.0.0.1:43540 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:16:42.927397 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.926358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.0.98.61:34049)
0927 21:16:42.926405 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.926407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.926753 (+ 346us) server_negotiation.cc:407] Connection header received
0927 21:16:42.926861 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.926862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.926876 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.926908 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.927328 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43548: BlockingRecv error: recv got EOF from 127.0.0.1:43548 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:16:42.945832 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.944798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.0.98.61:34049)
0927 21:16:42.944843 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.944845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.945138 (+ 293us) server_negotiation.cc:407] Connection header received
0927 21:16:42.945241 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.945242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.945254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.945318 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.945766 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43560: BlockingRecv error: recv got EOF from 127.0.0.1:43560 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:16:42.980118 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:42.979172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.0.98.61:34049)
0927 21:16:42.979213 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:42.979215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:42.979483 (+ 268us) server_negotiation.cc:407] Connection header received
0927 21:16:42.979598 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:42.979599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:42.979611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:42.979640 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:42.980053 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43564: BlockingRecv error: recv got EOF from 127.0.0.1:43564 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 21:16:43.046126 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:43.045473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.0.98.61:34049)
0927 21:16:43.045533 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:43.045536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:43.045666 (+ 130us) server_negotiation.cc:407] Connection header received
0927 21:16:43.045720 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:43.045721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:43.045731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:43.045778 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:43.046064 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43578: BlockingRecv error: recv got EOF from 127.0.0.1:43578 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 21:16:43.175967 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:43.175337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.0.98.61:34049)
0927 21:16:43.175424 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:43.175426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:43.175438 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:16:43.175532 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:43.175533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:43.175544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:43.175585 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:43.175917 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43590: BlockingRecv error: recv got EOF from 127.0.0.1:43590 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:16:43.257797 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3895d31aec94c22af5f1faa44bd4a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } peers { permanent_uuid: "93f2e85fa5264c6f9af1950c51238cb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34049 } } peers { permanent_uuid: "efdac131313e4599bd357031fa15363b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35443 } } }
I20260927 21:16:43.257961 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3895d31aec94c22af5f1faa44bd4a2e [sys.catalog]: This master's current role is: LEADER
W20260927 21:16:43.433874 23111 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:16:43.433334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.0.98.61:34049)
0927 21:16:43.433397 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:43.433399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:43.433408 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:43.433433 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:43.433434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:43.433444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:43.433505 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:16:43.433812 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43602: BlockingRecv error: recv got EOF from 127.0.0.1:43602 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:16:43.806464 23201 catalog_manager.cc:1259] Loaded cluster ID: 55945055f53c49a6b35de404868541c3
I20260927 21:16:43.806573 23201 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b: loading cluster ID for follower catalog manager: success
I20260927 21:16:43.807337 23201 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b: acquiring CA information for follower catalog manager: success
I20260927 21:16:43.807618 23201 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efdac131313e4599bd357031fa15363b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:16:43.868049 23244 catalog_manager.cc:1259] Loaded cluster ID: 55945055f53c49a6b35de404868541c3
I20260927 21:16:43.868134 23244 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0: loading cluster ID for follower catalog manager: success
I20260927 21:16:43.868892 23244 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0: acquiring CA information for follower catalog manager: success
I20260927 21:16:43.869151 23244 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93f2e85fa5264c6f9af1950c51238cb0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:16:43.949859 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:43.947160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.0.98.61:34049)
0927 21:16:43.947260 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:43.947262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:43.947271 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:16:43.947333 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:43.947334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:43.947394 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:43.947468 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:43.947701 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.948217 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.949004 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.949204 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.949250 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:43.949496 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:43.949615 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:43.949677 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:43.949757 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 21:16:43.949777 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
I20260927 21:16:43.953869 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:43.951529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.0.98.60:35443)
0927 21:16:43.951678 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:43.951680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:43.951688 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:43.951722 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:43.951722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:43.951756 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:43.951842 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:43.951922 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.952402 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.953138 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.953369 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.953427 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:43.953573 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:43.953687 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:43.953745 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:43.953759 (+ 14us) server_negotiation.cc:299] Negotiation successful
0927 21:16:43.953779 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
I20260927 21:16:43.958040 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:43.955603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.0.98.60:35443)
0927 21:16:43.955645 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:43.955647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:43.955655 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:16:43.955698 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:43.955699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:43.955710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:43.955753 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:43.955876 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.956438 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.957312 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.957614 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.957680 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:43.957805 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:43.957839 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:43.957881 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:43.957936 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 21:16:43.957959 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:16:43.958573 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:43.955749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.0.98.61:34049)
0927 21:16:43.955788 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:43.955790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:43.955897 (+ 107us) server_negotiation.cc:407] Connection header received
0927 21:16:43.955930 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:43.955930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:43.955942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:43.955994 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:43.956122 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.956870 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.957745 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.958031 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.958086 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:43.958327 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:43.958361 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:43.958404 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:43.958463 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 21:16:43.958488 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:16:43.958976 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:16:43.955621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.0.98.62:34847)
0927 21:16:43.955740 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 21:16:43.955742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:16:43.955824 (+ 82us) server_negotiation.cc:407] Connection header received
0927 21:16:43.955870 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:16:43.955871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:16:43.955923 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:16:43.956001 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:16:43.956160 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.956902 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.957885 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:16:43.958283 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:16:43.958343 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:16:43.958632 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:16:43.958754 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:16:43.958812 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:16:43.958860 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 21:16:43.958891 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260927 21:16:43.964457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 22964
I20260927 21:16:43.969218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 23039
I20260927 21:16:43.973789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm7AuFZ/build/release/bin/kudu with pid 23114
2026-09-27T21:16:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2752 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5503 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274250 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST